* [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-24 10:32 ` Christian Borntraeger
2018-09-12 19:42 ` [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART Tony Krowiak
` (26 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, David Hildenbrand, Tony Krowiak
From: David Hildenbrand <david@redhat.com>
VCPU requests and VCPU blocking right now don't take care of the vSIE
(as it was not necessary until now). But we want to have synchronous VCPU
requests that will also be handled before running the vSIE again.
So let's simulate a SIE entry of the VCPU when calling the sie during
vSIE handling and check for PROG_ flags. The existing infrastructure
(e.g. exit_sie()) will then detect that the SIE (in form of the vSIE) is
running and properly kick the vSIE CPU, resulting in it leaving the vSIE
loop and therefore the vSIE interception handler, allowing it to handle
VCPU requests.
E.g. if we want to modify the crycb of the VCPU and make sure that any
masks also get applied to the VSIE crycb shadow (which uses masks from the
VCPU crycb), we will need a way to hinder the vSIE from running and make
sure to process the updated crycb before reentering the vSIE again.
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
arch/s390/kvm/kvm-s390.c | 9 ++++++++-
arch/s390/kvm/kvm-s390.h | 1 +
arch/s390/kvm/vsie.c | 21 +++++++++++++++++----
3 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index f69333f..0b5aff0 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2768,18 +2768,25 @@ static void kvm_s390_vcpu_request(struct kvm_vcpu *vcpu)
exit_sie(vcpu);
}
+bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu)
+{
+ return atomic_read(&vcpu->arch.sie_block->prog20) &
+ (PROG_BLOCK_SIE | PROG_REQUEST);
+}
+
static void kvm_s390_vcpu_request_handled(struct kvm_vcpu *vcpu)
{
atomic_andnot(PROG_REQUEST, &vcpu->arch.sie_block->prog20);
}
/*
- * Kick a guest cpu out of SIE and wait until SIE is not running.
+ * Kick a guest cpu out of (v)SIE and wait until (v)SIE is not running.
* If the CPU is not running (e.g. waiting as idle) the function will
* return immediately. */
void exit_sie(struct kvm_vcpu *vcpu)
{
kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOP_INT);
+ kvm_s390_vsie_kick(vcpu);
while (vcpu->arch.sie_block->prog0c & PROG_IN_SIE)
cpu_relax();
}
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 981e3ba..1f6e36c 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -290,6 +290,7 @@ void kvm_s390_set_tod_clock(struct kvm *kvm,
void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);
void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);
void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);
+bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu);
void exit_sie(struct kvm_vcpu *vcpu);
void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);
int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index a2b28cd..12b9707 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -830,7 +830,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
int guest_bp_isolation;
- int rc;
+ int rc = 0;
handle_last_fault(vcpu, vsie_page);
@@ -858,7 +858,18 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
guest_enter_irqoff();
local_irq_enable();
- rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
+ /*
+ * Simulate a SIE entry of the VCPU (see sie64a), so VCPU blocking
+ * and VCPU requests also hinder the vSIE from running and lead
+ * to an immediate exit. kvm_s390_vsie_kick() has to be used to
+ * also kick the vSIE.
+ */
+ vcpu->arch.sie_block->prog0c |= PROG_IN_SIE;
+ barrier();
+ if (!kvm_s390_vcpu_sie_inhibited(vcpu))
+ rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
+ barrier();
+ vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
local_irq_disable();
guest_exit_irqoff();
@@ -1005,7 +1016,8 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
if (rc == -EAGAIN)
rc = 0;
if (rc || scb_s->icptcode || signal_pending(current) ||
- kvm_s390_vcpu_has_irq(vcpu, 0))
+ kvm_s390_vcpu_has_irq(vcpu, 0) ||
+ kvm_s390_vcpu_sie_inhibited(vcpu))
break;
}
@@ -1122,7 +1134,8 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu)
if (unlikely(scb_addr & 0x1ffUL))
return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
- if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0))
+ if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
+ kvm_s390_vcpu_sie_inhibited(vcpu))
return 0;
vsie_page = get_vsie_page(vcpu->kvm, scb_addr);
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit
2018-09-12 19:42 ` [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit Tony Krowiak
@ 2018-09-24 10:32 ` Christian Borntraeger
2018-09-24 16:53 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Christian Borntraeger @ 2018-09-24 10:32 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja,
David Hildenbrand, Tony Krowiak
On 09/12/2018 09:42 PM, Tony Krowiak wrote:
> From: David Hildenbrand <david@redhat.com>
>
> VCPU requests and VCPU blocking right now don't take care of the vSIE
> (as it was not necessary until now). But we want to have synchronous VCPU
> requests that will also be handled before running the vSIE again.
>
> So let's simulate a SIE entry of the VCPU when calling the sie during
> vSIE handling and check for PROG_ flags. The existing infrastructure
> (e.g. exit_sie()) will then detect that the SIE (in form of the vSIE) is
> running and properly kick the vSIE CPU, resulting in it leaving the vSIE
> loop and therefore the vSIE interception handler, allowing it to handle
> VCPU requests.
>
> E.g. if we want to modify the crycb of the VCPU and make sure that any
> masks also get applied to the VSIE crycb shadow (which uses masks from the
> VCPU crycb), we will need a way to hinder the vSIE from running and make
> sure to process the updated crycb before reentering the vSIE again.
>
> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
I guess you should want to swap both signoffs? In the end they should show the
patch flow.
Other than that:
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
> arch/s390/kvm/kvm-s390.h | 1 +
> arch/s390/kvm/vsie.c | 21 +++++++++++++++++----
> 3 files changed, 26 insertions(+), 5 deletions(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index f69333f..0b5aff0 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -2768,18 +2768,25 @@ static void kvm_s390_vcpu_request(struct kvm_vcpu *vcpu)
> exit_sie(vcpu);
> }
>
> +bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu)
> +{
> + return atomic_read(&vcpu->arch.sie_block->prog20) &
> + (PROG_BLOCK_SIE | PROG_REQUEST);
> +}
> +
> static void kvm_s390_vcpu_request_handled(struct kvm_vcpu *vcpu)
> {
> atomic_andnot(PROG_REQUEST, &vcpu->arch.sie_block->prog20);
> }
>
> /*
> - * Kick a guest cpu out of SIE and wait until SIE is not running.
> + * Kick a guest cpu out of (v)SIE and wait until (v)SIE is not running.
> * If the CPU is not running (e.g. waiting as idle) the function will
> * return immediately. */
> void exit_sie(struct kvm_vcpu *vcpu)
> {
> kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOP_INT);
> + kvm_s390_vsie_kick(vcpu);
> while (vcpu->arch.sie_block->prog0c & PROG_IN_SIE)
> cpu_relax();
> }
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 981e3ba..1f6e36c 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -290,6 +290,7 @@ void kvm_s390_set_tod_clock(struct kvm *kvm,
> void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);
> void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);
> void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);
> +bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu);
> void exit_sie(struct kvm_vcpu *vcpu);
> void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);
> int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);
> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
> index a2b28cd..12b9707 100644
> --- a/arch/s390/kvm/vsie.c
> +++ b/arch/s390/kvm/vsie.c
> @@ -830,7 +830,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
> struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
> int guest_bp_isolation;
> - int rc;
> + int rc = 0;
>
> handle_last_fault(vcpu, vsie_page);
>
> @@ -858,7 +858,18 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> guest_enter_irqoff();
> local_irq_enable();
>
> - rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
> + /*
> + * Simulate a SIE entry of the VCPU (see sie64a), so VCPU blocking
> + * and VCPU requests also hinder the vSIE from running and lead
> + * to an immediate exit. kvm_s390_vsie_kick() has to be used to
> + * also kick the vSIE.
> + */
> + vcpu->arch.sie_block->prog0c |= PROG_IN_SIE;
> + barrier();
> + if (!kvm_s390_vcpu_sie_inhibited(vcpu))
> + rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
> + barrier();
> + vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
>
> local_irq_disable();
> guest_exit_irqoff();
> @@ -1005,7 +1016,8 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> if (rc == -EAGAIN)
> rc = 0;
> if (rc || scb_s->icptcode || signal_pending(current) ||
> - kvm_s390_vcpu_has_irq(vcpu, 0))
> + kvm_s390_vcpu_has_irq(vcpu, 0) ||
> + kvm_s390_vcpu_sie_inhibited(vcpu))
> break;
> }
>
> @@ -1122,7 +1134,8 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu)
> if (unlikely(scb_addr & 0x1ffUL))
> return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>
> - if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0))
> + if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
> + kvm_s390_vcpu_sie_inhibited(vcpu))
> return 0;
>
> vsie_page = get_vsie_page(vcpu->kvm, scb_addr);
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit
2018-09-24 10:32 ` Christian Borntraeger
@ 2018-09-24 16:53 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:53 UTC (permalink / raw)
To: Christian Borntraeger, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja,
David Hildenbrand
On 09/24/2018 06:32 AM, Christian Borntraeger wrote:
>
>
> On 09/12/2018 09:42 PM, Tony Krowiak wrote:
>> From: David Hildenbrand <david@redhat.com>
>>
>> VCPU requests and VCPU blocking right now don't take care of the vSIE
>> (as it was not necessary until now). But we want to have synchronous VCPU
>> requests that will also be handled before running the vSIE again.
>>
>> So let's simulate a SIE entry of the VCPU when calling the sie during
>> vSIE handling and check for PROG_ flags. The existing infrastructure
>> (e.g. exit_sie()) will then detect that the SIE (in form of the vSIE) is
>> running and properly kick the vSIE CPU, resulting in it leaving the vSIE
>> loop and therefore the vSIE interception handler, allowing it to handle
>> VCPU requests.
>>
>> E.g. if we want to modify the crycb of the VCPU and make sure that any
>> masks also get applied to the VSIE crycb shadow (which uses masks from the
>> VCPU crycb), we will need a way to hinder the vSIE from running and make
>> sure to process the updated crycb before reentering the vSIE again.
>>
>> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>
> I guess you should want to swap both signoffs? In the end they should show the
> patch flow.
Will do.
>
> Other than that:
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
>
>> ---
>> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
>> arch/s390/kvm/kvm-s390.h | 1 +
>> arch/s390/kvm/vsie.c | 21 +++++++++++++++++----
>> 3 files changed, 26 insertions(+), 5 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index f69333f..0b5aff0 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -2768,18 +2768,25 @@ static void kvm_s390_vcpu_request(struct kvm_vcpu *vcpu)
>> exit_sie(vcpu);
>> }
>>
>> +bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu)
>> +{
>> + return atomic_read(&vcpu->arch.sie_block->prog20) &
>> + (PROG_BLOCK_SIE | PROG_REQUEST);
>> +}
>> +
>> static void kvm_s390_vcpu_request_handled(struct kvm_vcpu *vcpu)
>> {
>> atomic_andnot(PROG_REQUEST, &vcpu->arch.sie_block->prog20);
>> }
>>
>> /*
>> - * Kick a guest cpu out of SIE and wait until SIE is not running.
>> + * Kick a guest cpu out of (v)SIE and wait until (v)SIE is not running.
>> * If the CPU is not running (e.g. waiting as idle) the function will
>> * return immediately. */
>> void exit_sie(struct kvm_vcpu *vcpu)
>> {
>> kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOP_INT);
>> + kvm_s390_vsie_kick(vcpu);
>> while (vcpu->arch.sie_block->prog0c & PROG_IN_SIE)
>> cpu_relax();
>> }
>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>> index 981e3ba..1f6e36c 100644
>> --- a/arch/s390/kvm/kvm-s390.h
>> +++ b/arch/s390/kvm/kvm-s390.h
>> @@ -290,6 +290,7 @@ void kvm_s390_set_tod_clock(struct kvm *kvm,
>> void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);
>> void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);
>> void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);
>> +bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu);
>> void exit_sie(struct kvm_vcpu *vcpu);
>> void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);
>> int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);
>> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
>> index a2b28cd..12b9707 100644
>> --- a/arch/s390/kvm/vsie.c
>> +++ b/arch/s390/kvm/vsie.c
>> @@ -830,7 +830,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
>> struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
>> int guest_bp_isolation;
>> - int rc;
>> + int rc = 0;
>>
>> handle_last_fault(vcpu, vsie_page);
>>
>> @@ -858,7 +858,18 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> guest_enter_irqoff();
>> local_irq_enable();
>>
>> - rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
>> + /*
>> + * Simulate a SIE entry of the VCPU (see sie64a), so VCPU blocking
>> + * and VCPU requests also hinder the vSIE from running and lead
>> + * to an immediate exit. kvm_s390_vsie_kick() has to be used to
>> + * also kick the vSIE.
>> + */
>> + vcpu->arch.sie_block->prog0c |= PROG_IN_SIE;
>> + barrier();
>> + if (!kvm_s390_vcpu_sie_inhibited(vcpu))
>> + rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
>> + barrier();
>> + vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
>>
>> local_irq_disable();
>> guest_exit_irqoff();
>> @@ -1005,7 +1016,8 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> if (rc == -EAGAIN)
>> rc = 0;
>> if (rc || scb_s->icptcode || signal_pending(current) ||
>> - kvm_s390_vcpu_has_irq(vcpu, 0))
>> + kvm_s390_vcpu_has_irq(vcpu, 0) ||
>> + kvm_s390_vcpu_sie_inhibited(vcpu))
>> break;
>> }
>>
>> @@ -1122,7 +1134,8 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu)
>> if (unlikely(scb_addr & 0x1ffUL))
>> return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>>
>> - if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0))
>> + if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
>> + kvm_s390_vcpu_sie_inhibited(vcpu))
>> return 0;
>>
>> vsie_page = get_vsie_page(vcpu->kvm, scb_addr);
>>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
2018-09-12 19:42 ` [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-24 10:49 ` Christian Borntraeger
2018-09-12 19:42 ` [PATCH v10 03/26] KVM: s390: refactor crypto initialization Tony Krowiak
` (25 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, David Hildenbrand, Tony Krowiak
From: David Hildenbrand <david@redhat.com>
When we change the crycb (or execution controls), we also have to make sure
that the vSIE shadow datastructures properly consider the changed
values before rerunning the vSIE. We can achieve that by simply using a
VCPU request now.
This has to be a synchronous request (== handled before entering the
(v)SIE again).
The request will make sure that the vSIE handler is left, and that the
request will be processed (NOP), therefore forcing a reload of all
vSIE data (including rebuilding the crycb) when re-entering the vSIE
interception handler the next time.
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/kvm/kvm-s390.c | 7 ++++++-
2 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 29c940b..75d3962 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -44,6 +44,7 @@
#define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2)
#define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)
#define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4)
+#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5)
#define SIGP_CTRL_C 0x80
#define SIGP_CTRL_SCN_MASK 0x3f
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 0b5aff0..876fbb2 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -844,8 +844,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
kvm_s390_vcpu_block_all(kvm);
- kvm_for_each_vcpu(i, vcpu, kvm)
+ kvm_for_each_vcpu(i, vcpu, kvm) {
kvm_s390_vcpu_crypto_setup(vcpu);
+ /* recreate the shadow crycb by leaving the VSIE handler */
+ kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+ }
kvm_s390_vcpu_unblock_all(kvm);
}
@@ -3203,6 +3206,8 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
/* nothing to do, just clear the request */
kvm_clear_request(KVM_REQ_UNHALT, vcpu);
+ /* we left the vsie handler, nothing to do, just clear the request */
+ kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu);
return 0;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
2018-09-12 19:42 ` [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART Tony Krowiak
@ 2018-09-24 10:49 ` Christian Borntraeger
2018-09-24 16:48 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Christian Borntraeger @ 2018-09-24 10:49 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja,
David Hildenbrand, Tony Krowiak
On 09/12/2018 09:42 PM, Tony Krowiak wrote:
> From: David Hildenbrand <david@redhat.com>
>
> When we change the crycb (or execution controls), we also have to make sure
> that the vSIE shadow datastructures properly consider the changed
> values before rerunning the vSIE. We can achieve that by simply using a
> VCPU request now.
>
> This has to be a synchronous request (== handled before entering the
> (v)SIE again).
>
> The request will make sure that the vSIE handler is left, and that the
> request will be processed (NOP), therefore forcing a reload of all
> vSIE data (including rebuilding the crycb) when re-entering the vSIE
> interception handler the next time.
>
> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
same here? First David then Tomy?
otherwise
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-s390.c | 7 ++++++-
> 2 files changed, 7 insertions(+), 1 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 29c940b..75d3962 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -44,6 +44,7 @@
> #define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2)
> #define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)
> #define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4)
> +#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5)
>
> #define SIGP_CTRL_C 0x80
> #define SIGP_CTRL_SCN_MASK 0x3f
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 0b5aff0..876fbb2 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -844,8 +844,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>
> kvm_s390_vcpu_block_all(kvm);
>
> - kvm_for_each_vcpu(i, vcpu, kvm)
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> kvm_s390_vcpu_crypto_setup(vcpu);
> + /* recreate the shadow crycb by leaving the VSIE handler */
> + kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
> + }
>
> kvm_s390_vcpu_unblock_all(kvm);
> }
> @@ -3203,6 +3206,8 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
>
> /* nothing to do, just clear the request */
> kvm_clear_request(KVM_REQ_UNHALT, vcpu);
> + /* we left the vsie handler, nothing to do, just clear the request */
> + kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu);
>
> return 0;
> }
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
2018-09-24 10:49 ` Christian Borntraeger
@ 2018-09-24 16:48 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:48 UTC (permalink / raw)
To: Christian Borntraeger, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja,
David Hildenbrand
On 09/24/2018 06:49 AM, Christian Borntraeger wrote:
> On 09/12/2018 09:42 PM, Tony Krowiak wrote:
>> From: David Hildenbrand <david@redhat.com>
>>
>> When we change the crycb (or execution controls), we also have to make sure
>> that the vSIE shadow datastructures properly consider the changed
>> values before rerunning the vSIE. We can achieve that by simply using a
>> VCPU request now.
>>
>> This has to be a synchronous request (== handled before entering the
>> (v)SIE again).
>>
>> The request will make sure that the vSIE handler is left, and that the
>> request will be processed (NOP), therefore forcing a reload of all
>> vSIE data (including rebuilding the crycb) when re-entering the vSIE
>> interception handler the next time.
>>
>> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>
>
> same here? First David then Tomy?
Will do.
>
> otherwise
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
>
>> ---
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 7 ++++++-
>> 2 files changed, 7 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 29c940b..75d3962 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -44,6 +44,7 @@
>> #define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2)
>> #define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)
>> #define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4)
>> +#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5)
>>
>> #define SIGP_CTRL_C 0x80
>> #define SIGP_CTRL_SCN_MASK 0x3f
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 0b5aff0..876fbb2 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -844,8 +844,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>
>> kvm_s390_vcpu_block_all(kvm);
>>
>> - kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> kvm_s390_vcpu_crypto_setup(vcpu);
>> + /* recreate the shadow crycb by leaving the VSIE handler */
>> + kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
>> + }
>>
>> kvm_s390_vcpu_unblock_all(kvm);
>> }
>> @@ -3203,6 +3206,8 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
>>
>> /* nothing to do, just clear the request */
>> kvm_clear_request(KVM_REQ_UNHALT, vcpu);
>> + /* we left the vsie handler, nothing to do, just clear the request */
>> + kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu);
>>
>> return 0;
>> }
>>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 03/26] KVM: s390: refactor crypto initialization
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
2018-09-12 19:42 ` [PATCH v10 01/26] KVM: s390: vsie: simulate VCPU SIE entry/exit Tony Krowiak
2018-09-12 19:42 ` [PATCH v10 02/26] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-21 23:18 ` Tony Krowiak
2018-09-12 19:42 ` [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
` (24 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
This patch refactors the code that initializes and sets up the
crypto configuration for a guest. The following changes are
implemented via this patch:
1. Prior to the introduction of AP device virtualization, it
was not necessary to provide guest access to the CRYCB
unless the MSA extension 3 (MSAX3) facility was installed
on the host system. With the introduction of AP device
virtualization, the CRYCB must be made accessible to the
guest if the AP instructions are installed on the host
and are to be provided to the guest.
2. Introduces a flag indicating AP instructions executed on
the guest shall be interpreted by the firmware. It is
initialized to indicate AP instructions are to be
to be interpreted and is used to set the SIE bit for
each vcpu during vcpu setup.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
arch/s390/include/asm/kvm_host.h | 2 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/kvm-s390.c | 71 ++++++++++++++++++--------------------
3 files changed, 37 insertions(+), 37 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 75d3962..423cce7 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
#define ECA_AIV 0x00200000
#define ECA_VX 0x00020000
#define ECA_PROTEXCI 0x00002000
+#define ECA_APIE 0x00000008
#define ECA_SII 0x00000001
__u32 eca; /* 0x004c */
#define ICPT_INST 0x04
@@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
__u8 reservede4[4]; /* 0x00e4 */
__u64 tecmc; /* 0x00e8 */
__u8 reservedf0[12]; /* 0x00f0 */
+#define CRYCB_FORMAT_MASK 0x00000003
#define CRYCB_FORMAT1 0x00000001
#define CRYCB_FORMAT2 0x00000003
__u32 crycbd; /* 0x00fc */
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index 9a50f02..8c23afc 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
#define KVM_S390_VM_CPU_FEAT_KSS 13
+#define KVM_S390_VM_CPU_FEAT_AP 14
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 876fbb2..d717041 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -40,6 +40,7 @@
#include <asm/sclp.h>
#include <asm/cpacf.h>
#include <asm/timex.h>
+#include <asm/ap.h>
#include "kvm-s390.h"
#include "gaccess.h"
@@ -1995,49 +1996,37 @@ long kvm_arch_vm_ioctl(struct file *filp,
return r;
}
-static int kvm_s390_query_ap_config(u8 *config)
-{
- u32 fcn_code = 0x04000000UL;
- u32 cc = 0;
-
- memset(config, 0, 128);
- asm volatile(
- "lgr 0,%1\n"
- "lgr 2,%2\n"
- ".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: ipm %0\n"
- "srl %0,28\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+r" (cc)
- : "r" (fcn_code), "r" (config)
- : "cc", "0", "2", "memory"
- );
-
- return cc;
-}
-
static int kvm_s390_apxa_installed(void)
{
- u8 config[128];
- int cc;
+ struct ap_config_info info;
- if (test_facility(12)) {
- cc = kvm_s390_query_ap_config(config);
-
- if (cc)
- pr_err("PQAP(QCI) failed with cc=%d", cc);
- else
- return config[0] & 0x40;
+ if (ap_instructions_available()) {
+ if (ap_qci(&info) == 0)
+ return info.apxa;
}
return 0;
}
+/*
+ * The format of the crypto control block (CRYCB) is specified in the 3 low
+ * order bits of the CRYCB designation (CRYCBD) field as follows:
+ * Format 0: Neither the message security assist extension 3 (MSAX3) nor the
+ * AP extended addressing (APXA) facility are installed.
+ * Format 1: The APXA facility is not installed but the MSAX3 facility is.
+ * Format 2: Both the APXA and MSAX3 facilities are installed
+ */
static void kvm_s390_set_crycb_format(struct kvm *kvm)
{
kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
+ /* Clear the CRYCB format bits - i.e., set format 0 by default */
+ kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT_MASK);
+
+ /* Check whether MSAX3 is installed */
+ if (!test_kvm_facility(kvm, 76))
+ return;
+
if (kvm_s390_apxa_installed())
kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
else
@@ -2055,12 +2044,12 @@ static u64 kvm_s390_get_initial_cpuid(void)
static void kvm_s390_crypto_init(struct kvm *kvm)
{
- if (!test_kvm_facility(kvm, 76))
- return;
-
kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
kvm_s390_set_crycb_format(kvm);
+ if (!test_kvm_facility(kvm, 76))
+ return;
+
/* Enable AES/DEA protected key functions by default */
kvm->arch.crypto.aes_kw = 1;
kvm->arch.crypto.dea_kw = 1;
@@ -2586,17 +2575,25 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
{
- if (!test_kvm_facility(vcpu->kvm, 76))
+ /*
+ * If neither the AP instructions nor the MSAX3 facility are configured
+ * for the guest, there is nothing to set up.
+ */
+ if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
+ !test_kvm_facility(vcpu->kvm, 76))
return;
+ vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
+ if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
+ vcpu->arch.sie_block->eca |= ECA_APIE;
+
+ /* Set up protected key support */
if (vcpu->kvm->arch.crypto.aes_kw)
vcpu->arch.sie_block->ecb3 |= ECB3_AES;
if (vcpu->kvm->arch.crypto.dea_kw)
vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
-
- vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
}
void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 03/26] KVM: s390: refactor crypto initialization
2018-09-12 19:42 ` [PATCH v10 03/26] KVM: s390: refactor crypto initialization Tony Krowiak
@ 2018-09-21 23:18 ` Tony Krowiak
2018-09-24 8:35 ` David Hildenbrand
2018-09-24 10:34 ` Cornelia Huck
0 siblings, 2 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-21 23:18 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/12/2018 03:42 PM, Tony Krowiak wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> This patch refactors the code that initializes and sets up the
> crypto configuration for a guest. The following changes are
> implemented via this patch:
>
> 1. Prior to the introduction of AP device virtualization, it
> was not necessary to provide guest access to the CRYCB
> unless the MSA extension 3 (MSAX3) facility was installed
> on the host system. With the introduction of AP device
> virtualization, the CRYCB must be made accessible to the
> guest if the AP instructions are installed on the host
> and are to be provided to the guest.
>
> 2. Introduces a flag indicating AP instructions executed on
> the guest shall be interpreted by the firmware. It is
> initialized to indicate AP instructions are to be
> to be interpreted and is used to set the SIE bit for
> each vcpu during vcpu setup.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Acked-by: Janosch Frank <frankja@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/include/asm/kvm_host.h | 2 +
> arch/s390/include/uapi/asm/kvm.h | 1 +
> arch/s390/kvm/kvm-s390.c | 71 ++++++++++++++++++--------------------
> 3 files changed, 37 insertions(+), 37 deletions(-)
>
(...)
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index 9a50f02..8c23afc 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
> #define KVM_S390_VM_CPU_FEAT_KSS 13
> +#define KVM_S390_VM_CPU_FEAT_AP 14
> struct kvm_s390_vm_cpu_feat {
> __u64 feat[16];
> };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 876fbb2..d717041 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -40,6 +40,7 @@
> #include <asm/sclp.h>
> #include <asm/cpacf.h>
> #include <asm/timex.h>
> +#include <asm/ap.h>
> #include "kvm-s390.h"
> #include "gaccess.h"
>
(...)
> @@ -2586,17 +2575,25 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>
> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
> {
> - if (!test_kvm_facility(vcpu->kvm, 76))
> + /*
> + * If neither the AP instructions nor the MSAX3 facility are configured
> + * for the guest, there is nothing to set up.
> + */
> + if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
> + !test_kvm_facility(vcpu->kvm, 76))
> return;
>
> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>
> + if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
> + vcpu->arch.sie_block->eca |= ECA_APIE;
> +
> + /* Set up protected key support */
> if (vcpu->kvm->arch.crypto.aes_kw)
> vcpu->arch.sie_block->ecb3 |= ECB3_AES;
> if (vcpu->kvm->arch.crypto.dea_kw)
> vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
> -
> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> }
>
> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>
The fixup! patch below modifies this patch (03/26) to illustrate how
David's recommendation will be implemented for v11 of the series. It
is one of three fixup! patches (the other two are in responses to
11/26
and 25/26) included to generate discussion in v10 rather than
waiting until v11 for comments.
-----------------------------------8<-----------------------------------
From: Tony Krowiak <akrowiak@linux.ibm.com>
Date: Thu, 20 Sep 2018 12:26:08 -0400
Subject: [FIXUP v10] fixup!: KVM: s390: refactor crypto initialization
---
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/include/uapi/asm/kvm.h | 1 -
arch/s390/kvm/kvm-s390.c | 9 ++++-----
3 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h
b/arch/s390/include/asm/kvm_host.h
index 423cce7..79fa0a3 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -718,6 +718,7 @@ struct kvm_s390_crypto {
__u32 crycbd;
__u8 aes_kw;
__u8 dea_kw;
+ __u8 apie;
};
#define APCB0_MASK_SIZE 1
diff --git a/arch/s390/include/uapi/asm/kvm.h
b/arch/s390/include/uapi/asm/kvm.h
index 8c23afc..9a50f02 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -130,7 +130,6 @@ struct kvm_s390_vm_cpu_machine {
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
#define KVM_S390_VM_CPU_FEAT_KSS 13
-#define KVM_S390_VM_CPU_FEAT_AP 14
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index d717041..ae4769b 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2576,17 +2576,16 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
{
/*
- * If neither the AP instructions nor the MSAX3 facility are configured
- * for the guest, there is nothing to set up.
+ * If the AP instructions are not available and the MSAX3 facility
+ * is not configured for the guest, there is nothing to set up.
*/
- if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
- !test_kvm_facility(vcpu->kvm, 76))
+ if (!ap_instructions_available() && !test_kvm_facility(vcpu->kvm, 76))
return;
vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
- if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
+ if (vcpu->kvm->arch.crypto.apie)
vcpu->arch.sie_block->eca |= ECA_APIE;
/* Set up protected key support */
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 03/26] KVM: s390: refactor crypto initialization
2018-09-21 23:18 ` Tony Krowiak
@ 2018-09-24 8:35 ` David Hildenbrand
2018-09-24 10:34 ` Cornelia Huck
1 sibling, 0 replies; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 8:35 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 22/09/2018 01:18, Tony Krowiak wrote:
> On 09/12/2018 03:42 PM, Tony Krowiak wrote:
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> This patch refactors the code that initializes and sets up the
>> crypto configuration for a guest. The following changes are
>> implemented via this patch:
>>
>> 1. Prior to the introduction of AP device virtualization, it
>> was not necessary to provide guest access to the CRYCB
>> unless the MSA extension 3 (MSAX3) facility was installed
>> on the host system. With the introduction of AP device
>> virtualization, the CRYCB must be made accessible to the
>> guest if the AP instructions are installed on the host
>> and are to be provided to the guest.
>>
>> 2. Introduces a flag indicating AP instructions executed on
>> the guest shall be interpreted by the firmware. It is
>> initialized to indicate AP instructions are to be
>> to be interpreted and is used to set the SIE bit for
>> each vcpu during vcpu setup.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> Acked-by: Janosch Frank <frankja@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> arch/s390/include/asm/kvm_host.h | 2 +
>> arch/s390/include/uapi/asm/kvm.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 71 ++++++++++++++++++--------------------
>> 3 files changed, 37 insertions(+), 37 deletions(-)
>>
>
> (...)
>
>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>> index 9a50f02..8c23afc 100644
>> --- a/arch/s390/include/uapi/asm/kvm.h
>> +++ b/arch/s390/include/uapi/asm/kvm.h
>> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
>> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
>> #define KVM_S390_VM_CPU_FEAT_KSS 13
>> +#define KVM_S390_VM_CPU_FEAT_AP 14
>> struct kvm_s390_vm_cpu_feat {
>> __u64 feat[16];
>> };
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 876fbb2..d717041 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -40,6 +40,7 @@
>> #include <asm/sclp.h>
>> #include <asm/cpacf.h>
>> #include <asm/timex.h>
>> +#include <asm/ap.h>
>> #include "kvm-s390.h"
>> #include "gaccess.h"
>>
>
> (...)
>
>> @@ -2586,17 +2575,25 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>>
>> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
>> {
>> - if (!test_kvm_facility(vcpu->kvm, 76))
>> + /*
>> + * If neither the AP instructions nor the MSAX3 facility are configured
>> + * for the guest, there is nothing to set up.
>> + */
>> + if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
>> + !test_kvm_facility(vcpu->kvm, 76))
>> return;
>>
>> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>>
>> + if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>> +
>> + /* Set up protected key support */
>> if (vcpu->kvm->arch.crypto.aes_kw)
>> vcpu->arch.sie_block->ecb3 |= ECB3_AES;
>> if (vcpu->kvm->arch.crypto.dea_kw)
>> vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
>> -
>> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> }
>>
>> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>>
>
> The fixup! patch below modifies this patch (03/26) to illustrate how
>
> David's recommendation will be implemented for v11 of the series. It
>
> is one of three fixup! patches (the other two are in responses to
> 11/26
> and 25/26) included to generate discussion in v10 rather than
>
> waiting until v11 for comments.
>
> -----------------------------------8<-----------------------------------
>
> From: Tony Krowiak <akrowiak@linux.ibm.com>
> Date: Thu, 20 Sep 2018 12:26:08 -0400
> Subject: [FIXUP v10] fixup!: KVM: s390: refactor crypto initialization
>
> ---
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/include/uapi/asm/kvm.h | 1 -
> arch/s390/kvm/kvm-s390.c | 9 ++++-----
> 3 files changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h
> b/arch/s390/include/asm/kvm_host.h
> index 423cce7..79fa0a3 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -718,6 +718,7 @@ struct kvm_s390_crypto {
> __u32 crycbd;
> __u8 aes_kw;
> __u8 dea_kw;
> + __u8 apie;
> };
>
> #define APCB0_MASK_SIZE 1
> diff --git a/arch/s390/include/uapi/asm/kvm.h
> b/arch/s390/include/uapi/asm/kvm.h
> index 8c23afc..9a50f02 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -130,7 +130,6 @@ struct kvm_s390_vm_cpu_machine {
> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
> #define KVM_S390_VM_CPU_FEAT_KSS 13
> -#define KVM_S390_VM_CPU_FEAT_AP 14
> struct kvm_s390_vm_cpu_feat {
> __u64 feat[16];
> };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index d717041..ae4769b 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -2576,17 +2576,16 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
> {
> /*
> - * If neither the AP instructions nor the MSAX3 facility are configured
> - * for the guest, there is nothing to set up.
> + * If the AP instructions are not available and the MSAX3 facility
> + * is not configured for the guest, there is nothing to set up.
> */
> - if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
> - !test_kvm_facility(vcpu->kvm, 76))
> + if (!ap_instructions_available() && !test_kvm_facility(vcpu->kvm, 76))
> return;
>
> vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>
> - if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
> + if (vcpu->kvm->arch.crypto.apie)
> vcpu->arch.sie_block->eca |= ECA_APIE;
>
> /* Set up protected key support */
>
Looks sane, too.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 03/26] KVM: s390: refactor crypto initialization
2018-09-21 23:18 ` Tony Krowiak
2018-09-24 8:35 ` David Hildenbrand
@ 2018-09-24 10:34 ` Cornelia Huck
1 sibling, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 10:34 UTC (permalink / raw)
To: Tony Krowiak
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On Fri, 21 Sep 2018 19:18:57 -0400
Tony Krowiak <akrowiak@linux.ibm.com> wrote:
> On 09/12/2018 03:42 PM, Tony Krowiak wrote:
> > From: Tony Krowiak <akrowiak@linux.ibm.com>
> >
> > This patch refactors the code that initializes and sets up the
> > crypto configuration for a guest. The following changes are
> > implemented via this patch:
> >
> > 1. Prior to the introduction of AP device virtualization, it
> > was not necessary to provide guest access to the CRYCB
> > unless the MSA extension 3 (MSAX3) facility was installed
> > on the host system. With the introduction of AP device
> > virtualization, the CRYCB must be made accessible to the
> > guest if the AP instructions are installed on the host
> > and are to be provided to the guest.
> >
> > 2. Introduces a flag indicating AP instructions executed on
> > the guest shall be interpreted by the firmware. It is
> > initialized to indicate AP instructions are to be
> > to be interpreted and is used to set the SIE bit for
> > each vcpu during vcpu setup.
> >
> > Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> > Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> > Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > Acked-by: Janosch Frank <frankja@linux.ibm.com>
> > Tested-by: Michael Mueller <mimu@linux.ibm.com>
> > Tested-by: Farhan Ali <alifm@linux.ibm.com>
> > Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > ---
> > arch/s390/include/asm/kvm_host.h | 2 +
> > arch/s390/include/uapi/asm/kvm.h | 1 +
> > arch/s390/kvm/kvm-s390.c | 71 ++++++++++++++++++--------------------
> > 3 files changed, 37 insertions(+), 37 deletions(-)
(...)
> The fixup! patch below modifies this patch (03/26) to illustrate how
>
> David's recommendation will be implemented for v11 of the series. It
>
> is one of three fixup! patches (the other two are in responses to
> 11/26
> and 25/26) included to generate discussion in v10 rather than
>
> waiting until v11 for comments.
(...)
Looks good to me.
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (2 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 03/26] KVM: s390: refactor crypto initialization Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-20 15:31 ` Cornelia Huck
2018-09-12 19:42 ` [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
` (23 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces a new AP device driver. This device driver
is built on the VFIO mediated device framework. The framework
provides sysfs interfaces that facilitate passthrough
access by guests to devices installed on the linux host.
The VFIO AP device driver will serve two purposes:
1. Provide the interfaces to reserve AP devices for exclusive
use by KVM guests. This is accomplished by unbinding the
devices to be reserved for guest usage from the zcrypt
device driver and binding them to the VFIO AP device driver.
2. Implements the functions, callbacks and sysfs attribute
interfaces required to create one or more VFIO mediated
devices each of which will be used to configure the AP
matrix for a guest and serve as a file descriptor
for facilitating communication between QEMU and the
VFIO AP device driver.
When the VFIO AP device driver is initialized:
* It registers with the AP bus for control of type 10 (CEX4
and newer) AP queue devices. This limitation was imposed
due to:
1. A desire to keep the code as simple as possible;
2. Some older models are no longer supported by the kernel
and others are getting close to end of service.
The probe and remove callbacks will be provided to support
the binding/unbinding of AP queue devices to/from the VFIO
AP device driver.
* Creates a matrix device, /sys/devices/vfio_ap/matrix,
to serve as the parent of the mediated devices created, one
for each guest, and to hold the APQNs of the AP devices bound to
the VFIO AP device driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
MAINTAINERS | 10 +++
arch/s390/Kconfig | 11 +++
drivers/iommu/Kconfig | 8 ++
drivers/s390/crypto/Makefile | 4 +
drivers/s390/crypto/vfio_ap_drv.c | 134 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 34 ++++++++
6 files changed, 201 insertions(+), 0 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
create mode 100644 drivers/s390/crypto/vfio_ap_private.h
diff --git a/MAINTAINERS b/MAINTAINERS
index d870cb5..9284283 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12652,6 +12652,16 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: drivers/s390/crypto/
+S390 VFIO AP DRIVER
+M: Tony Krowiak <akrowiak@linux.ibm.com>
+M: Pierre Morel <pmorel@linux.ibm.com>
+M: Halil Pasic <pasic@linux.ibm.com>
+L: linux-s390@vger.kernel.org
+W: http://www.ibm.com/developerworks/linux/linux390/
+S: Supported
+F: drivers/s390/crypto/vfio_ap_drv.c
+F: drivers/s390/crypto/vfio_ap_private.h
+
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.ibm.com>
M: Benjamin Block <bblock@linux.ibm.com>
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 9a9c7a6..8cc8f25 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -773,6 +773,17 @@ config VFIO_CCW
To compile this driver as a module, choose M here: the
module will be called vfio_ccw.
+config VFIO_AP
+ def_tristate n
+ prompt "VFIO support for AP devices"
+ depends on S390_AP_IOMMU && VFIO_MDEV_DEVICE && KVM
+ help
+ This driver grants access to Adjunct Processor (AP) devices
+ via the VFIO mediated device interface.
+
+ To compile this driver as a module, choose M here: the module
+ will be called vfio_ap.
+
endmenu
menu "Dump support"
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index c60395b..83e6d99 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -372,6 +372,14 @@ config S390_CCW_IOMMU
Enables bits of IOMMU API required by VFIO. The iommu_ops
is not implemented as it is not necessary for VFIO.
+config S390_AP_IOMMU
+ bool "S390 AP IOMMU Support"
+ depends on S390 && ZCRYPT
+ select IOMMU_API
+ help
+ Enables bits of IOMMU API required by VFIO. The iommu_ops
+ is not implemented as it is not necessary for VFIO.
+
config MTK_IOMMU
bool "MTK IOMMU Support"
depends on ARM || ARM64
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index b59af54..48e466e 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
# pkey kernel module
pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
+
+# adjunct processor matrix
+vfio_ap-objs := vfio_ap_drv.o
+obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
new file mode 100644
index 0000000..3e635f0
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * VFIO based AP device driver
+ *
+ * Copyright IBM Corp. 2018
+ *
+ * Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include "vfio_ap_private.h"
+
+#define VFIO_AP_ROOT_NAME "vfio_ap"
+#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
+#define VFIO_AP_DEV_NAME "matrix"
+
+MODULE_AUTHOR("IBM Corporation");
+MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018");
+MODULE_LICENSE("GPL v2");
+
+static struct ap_driver vfio_ap_drv;
+
+static struct device_type vfio_ap_dev_type = {
+ .name = VFIO_AP_DEV_TYPE_NAME,
+};
+
+struct ap_matrix_dev *matrix_dev;
+
+/* Only type 10 adapters (CEX4 and later) are supported
+ * by the AP matrix device driver
+ */
+static struct ap_device_id ap_queue_ids[] = {
+ { .dev_type = AP_DEVICE_TYPE_CEX4,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX5,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX6,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { /* end of sibling */ },
+};
+
+MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
+
+static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
+{
+ return 0;
+}
+
+static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
+{
+ /* Nothing to do yet */
+}
+
+static void vfio_ap_matrix_dev_release(struct device *dev)
+{
+ struct ap_matrix_dev *matrix_dev = dev_get_drvdata(dev);
+
+ kfree(matrix_dev);
+}
+
+static int vfio_ap_matrix_dev_create(void)
+{
+ int ret;
+ struct device *root_device;
+
+ root_device = root_device_register(VFIO_AP_ROOT_NAME);
+ if (IS_ERR(root_device))
+ return PTR_ERR(root_device);
+
+ matrix_dev = kzalloc(sizeof(*matrix_dev), GFP_KERNEL);
+ if (!matrix_dev) {
+ ret = -ENOMEM;
+ goto matrix_alloc_err;
+ }
+
+ matrix_dev->device.type = &vfio_ap_dev_type;
+ dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
+ matrix_dev->device.parent = root_device;
+ matrix_dev->device.release = vfio_ap_matrix_dev_release;
+ matrix_dev->device.driver = &vfio_ap_drv.driver;
+
+ ret = device_register(&matrix_dev->device);
+ if (ret)
+ goto matrix_reg_err;
+
+ return 0;
+
+matrix_reg_err:
+ put_device(&matrix_dev->device);
+matrix_alloc_err:
+ root_device_unregister(root_device);
+
+ return ret;
+}
+
+static void vfio_ap_matrix_dev_destroy(void)
+{
+ device_unregister(&matrix_dev->device);
+ root_device_unregister(matrix_dev->device.parent);
+}
+
+int __init vfio_ap_init(void)
+{
+ int ret;
+
+ ret = vfio_ap_matrix_dev_create();
+ if (ret)
+ return ret;
+
+ memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
+ vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
+ vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
+ vfio_ap_drv.ids = ap_queue_ids;
+
+ ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
+ if (ret) {
+ vfio_ap_matrix_dev_destroy();
+ return ret;
+ }
+
+ return 0;
+}
+
+void __exit vfio_ap_exit(void)
+{
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy();
+}
+
+module_init(vfio_ap_init);
+module_exit(vfio_ap_exit);
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
new file mode 100644
index 0000000..6141420
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Private data and functions for adjunct processor VFIO matrix driver.
+ *
+ * Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
+ *
+ * Copyright IBM Corp. 2018
+ */
+
+#ifndef _VFIO_AP_PRIVATE_H_
+#define _VFIO_AP_PRIVATE_H_
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/mdev.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
+
+#include "ap_bus.h"
+
+#define VFIO_AP_MODULE_NAME "vfio_ap"
+#define VFIO_AP_DRV_NAME "vfio_ap"
+
+/**
+ * ap_matrix_dev - the AP matrix device structure
+ * @device: generic device structure associated with the AP matrix device
+ */
+struct ap_matrix_dev {
+ struct device device;
+};
+
+extern struct ap_matrix_dev *matrix_dev;
+
+#endif /* _VFIO_AP_PRIVATE_H_ */
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver
2018-09-12 19:42 ` [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
@ 2018-09-20 15:31 ` Cornelia Huck
2018-09-20 15:53 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-20 15:31 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:54 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces a new AP device driver. This device driver
> is built on the VFIO mediated device framework. The framework
> provides sysfs interfaces that facilitate passthrough
> access by guests to devices installed on the linux host.
>
> The VFIO AP device driver will serve two purposes:
>
> 1. Provide the interfaces to reserve AP devices for exclusive
> use by KVM guests. This is accomplished by unbinding the
> devices to be reserved for guest usage from the zcrypt
> device driver and binding them to the VFIO AP device driver.
>
> 2. Implements the functions, callbacks and sysfs attribute
> interfaces required to create one or more VFIO mediated
> devices each of which will be used to configure the AP
> matrix for a guest and serve as a file descriptor
> for facilitating communication between QEMU and the
> VFIO AP device driver.
>
> When the VFIO AP device driver is initialized:
>
> * It registers with the AP bus for control of type 10 (CEX4
> and newer) AP queue devices. This limitation was imposed
> due to:
>
> 1. A desire to keep the code as simple as possible;
>
> 2. Some older models are no longer supported by the kernel
> and others are getting close to end of service.
>
> The probe and remove callbacks will be provided to support
> the binding/unbinding of AP queue devices to/from the VFIO
> AP device driver.
>
> * Creates a matrix device, /sys/devices/vfio_ap/matrix,
> to serve as the parent of the mediated devices created, one
> for each guest, and to hold the APQNs of the AP devices bound to
> the VFIO AP device driver.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Christian's S-o-b looks a bit odd here; OTOH, it's probably him anyway
who will queue the patches in the end :)
> ---
> MAINTAINERS | 10 +++
> arch/s390/Kconfig | 11 +++
> drivers/iommu/Kconfig | 8 ++
> drivers/s390/crypto/Makefile | 4 +
> drivers/s390/crypto/vfio_ap_drv.c | 134 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 34 ++++++++
> 6 files changed, 201 insertions(+), 0 deletions(-)
> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver
2018-09-20 15:31 ` Cornelia Huck
@ 2018-09-20 15:53 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-20 15:53 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/20/2018 11:31 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:42:54 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces a new AP device driver. This device driver
>> is built on the VFIO mediated device framework. The framework
>> provides sysfs interfaces that facilitate passthrough
>> access by guests to devices installed on the linux host.
>>
>> The VFIO AP device driver will serve two purposes:
>>
>> 1. Provide the interfaces to reserve AP devices for exclusive
>> use by KVM guests. This is accomplished by unbinding the
>> devices to be reserved for guest usage from the zcrypt
>> device driver and binding them to the VFIO AP device driver.
>>
>> 2. Implements the functions, callbacks and sysfs attribute
>> interfaces required to create one or more VFIO mediated
>> devices each of which will be used to configure the AP
>> matrix for a guest and serve as a file descriptor
>> for facilitating communication between QEMU and the
>> VFIO AP device driver.
>>
>> When the VFIO AP device driver is initialized:
>>
>> * It registers with the AP bus for control of type 10 (CEX4
>> and newer) AP queue devices. This limitation was imposed
>> due to:
>>
>> 1. A desire to keep the code as simple as possible;
>>
>> 2. Some older models are no longer supported by the kernel
>> and others are getting close to end of service.
>>
>> The probe and remove callbacks will be provided to support
>> the binding/unbinding of AP queue devices to/from the VFIO
>> AP device driver.
>>
>> * Creates a matrix device, /sys/devices/vfio_ap/matrix,
>> to serve as the parent of the mediated devices created, one
>> for each guest, and to hold the APQNs of the AP devices bound to
>> the VFIO AP device driver.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Christian's S-o-b looks a bit odd here; OTOH, it's probably him anyway
> who will queue the patches in the end :)
It went in there when Christian submitted the patch series in my absence
back in
July while I was on vacation. I wasn't aware it was in the actual commit
message,
so it never got removed. I'll remove it if a v11 needs to be submitted.
>
>> ---
>> MAINTAINERS | 10 +++
>> arch/s390/Kconfig | 11 +++
>> drivers/iommu/Kconfig | 8 ++
>> drivers/s390/crypto/Makefile | 4 +
>> drivers/s390/crypto/vfio_ap_drv.c | 134 +++++++++++++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_private.h | 34 ++++++++
>> 6 files changed, 201 insertions(+), 0 deletions(-)
>> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (3 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 04/26] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-20 15:50 ` Cornelia Huck
2018-09-12 19:42 ` [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
` (22 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Registers the matrix device created by the VFIO AP device
driver with the VFIO mediated device framework.
Registering the matrix device will create the sysfs
structures needed to create mediated matrix devices
each of which will be used to configure the AP matrix
for a guest and connect it to the VFIO AP device driver.
Registering the matrix device with the VFIO mediated device
framework will create the following sysfs structures:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ create
To create a mediated device for the AP matrix device, write a UUID
to the create file:
uuidgen > create
A symbolic link to the mediated device's directory will be created in the
devices subdirectory named after the generated $uuid:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
............... [$uuid]
A symbolic link to the mediated device will also be created
in the vfio_ap matrix's directory:
/sys/devices/vfio_ap/matrix/[$uuid]
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
MAINTAINERS | 1 +
drivers/s390/crypto/Makefile | 2 +-
drivers/s390/crypto/vfio_ap_drv.c | 19 +++++
drivers/s390/crypto/vfio_ap_ops.c | 126 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 49 +++++++++++++
include/uapi/linux/vfio.h | 1 +
6 files changed, 197 insertions(+), 1 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 9284283..5a5aa0f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12661,6 +12661,7 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
+F: drivers/s390/crypto/vfio_ap_ops.c
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.ibm.com>
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index 48e466e..8d36b05 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -17,5 +17,5 @@ pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
# adjunct processor matrix
-vfio_ap-objs := vfio_ap_drv.o
+vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index 3e635f0..75f8bdc 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -76,6 +76,16 @@ static int vfio_ap_matrix_dev_create(void)
goto matrix_alloc_err;
}
+ /* Test if PQAP(QCI) instruction is available */
+ if (test_facility(12)) {
+ ret = ap_qci(&matrix_dev->info);
+ if (ret)
+ goto matrix_alloc_err;
+ }
+
+ mutex_init(&matrix_dev->lock);
+ INIT_LIST_HEAD(&matrix_dev->mdev_list);
+
matrix_dev->device.type = &vfio_ap_dev_type;
dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
matrix_dev->device.parent = root_device;
@@ -121,11 +131,20 @@ int __init vfio_ap_init(void)
return ret;
}
+ ret = vfio_ap_mdev_register();
+ if (ret) {
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy();
+
+ return ret;
+ }
+
return 0;
}
void __exit vfio_ap_exit(void)
{
+ vfio_ap_mdev_unregister();
ap_driver_unregister(&vfio_ap_drv);
vfio_ap_matrix_dev_destroy();
}
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
new file mode 100644
index 0000000..99ed303
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -0,0 +1,126 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Adjunct processor matrix VFIO device driver callbacks.
+ *
+ * Copyright IBM Corp. 2018
+ *
+ * Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
+ * Halil Pasic <pasic@linux.ibm.com>
+ * Pierre Morel <pmorel@linux.ibm.com>
+ */
+#include <linux/string.h>
+#include <linux/vfio.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/ctype.h>
+#include <asm/zcrypt.h>
+
+#include "vfio_ap_private.h"
+
+#define VFIO_AP_MDEV_TYPE_HWVIRT "passthrough"
+#define VFIO_AP_MDEV_NAME_HWVIRT "VFIO AP Passthrough Device"
+
+static void vfio_ap_matrix_init(struct ap_config_info *info,
+ struct ap_matrix *matrix)
+{
+ matrix->apm_max = info->apxa ? info->Na : 63;
+ matrix->aqm_max = info->apxa ? info->Nd : 15;
+ matrix->adm_max = info->apxa ? info->Nd : 15;
+}
+
+static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev;
+
+ if ((atomic_dec_if_positive(&matrix_dev->available_instances) < 0))
+ return -EPERM;
+
+ matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
+ if (!matrix_mdev) {
+ atomic_inc(&matrix_dev->available_instances);
+ return -ENOMEM;
+ }
+
+ vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
+ mdev_set_drvdata(mdev, matrix_mdev);
+ mutex_lock(&matrix_dev->lock);
+ list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
+ mutex_unlock(&matrix_dev->lock);
+
+ return 0;
+}
+
+static int vfio_ap_mdev_remove(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ mutex_lock(&matrix_dev->lock);
+ list_del(&matrix_mdev->node);
+ mutex_unlock(&matrix_dev->lock);
+
+ kfree(matrix_mdev);
+ mdev_set_drvdata(mdev, NULL);
+ atomic_inc(&matrix_dev->available_instances);
+
+ return 0;
+}
+
+static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
+}
+
+MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t available_instances_show(struct kobject *kobj,
+ struct device *dev, char *buf)
+{
+ return sprintf(buf, "%d\n",
+ atomic_read(&matrix_dev->available_instances));
+}
+
+MDEV_TYPE_ATTR_RO(available_instances);
+
+static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
+ char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_DEVICE_API_AP_STRING);
+}
+
+MDEV_TYPE_ATTR_RO(device_api);
+
+static struct attribute *vfio_ap_mdev_type_attrs[] = {
+ &mdev_type_attr_name.attr,
+ &mdev_type_attr_device_api.attr,
+ &mdev_type_attr_available_instances.attr,
+ NULL,
+};
+
+static struct attribute_group vfio_ap_mdev_hwvirt_type_group = {
+ .name = VFIO_AP_MDEV_TYPE_HWVIRT,
+ .attrs = vfio_ap_mdev_type_attrs,
+};
+
+static struct attribute_group *vfio_ap_mdev_type_groups[] = {
+ &vfio_ap_mdev_hwvirt_type_group,
+ NULL,
+};
+
+static const struct mdev_parent_ops vfio_ap_matrix_ops = {
+ .owner = THIS_MODULE,
+ .supported_type_groups = vfio_ap_mdev_type_groups,
+ .create = vfio_ap_mdev_create,
+ .remove = vfio_ap_mdev_remove,
+};
+
+int vfio_ap_mdev_register(void)
+{
+ atomic_set(&matrix_dev->available_instances, MAX_ZDEV_ENTRIES_EXT);
+
+ return mdev_register_device(&matrix_dev->device, &vfio_ap_matrix_ops);
+}
+
+void vfio_ap_mdev_unregister(void)
+{
+ mdev_unregister_device(&matrix_dev->device);
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 6141420..a2eab78 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -3,6 +3,7 @@
* Private data and functions for adjunct processor VFIO matrix driver.
*
* Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
+ * Halil Pasic <pasic@linux.ibm.com>
*
* Copyright IBM Corp. 2018
*/
@@ -24,11 +25,59 @@
/**
* ap_matrix_dev - the AP matrix device structure
* @device: generic device structure associated with the AP matrix device
+ * @available_instances: number of mediated matrix devices that can be created
+ * @info: the struct containing the output from the PQAP(TAPQ) instruction
+ * mdev_list: the list of mediated matrix devices created
+ * lock: mutex for locking the AP matrix device. This lock will be
+ * taken every time we fiddle with state managed by the vfio_ap
+ * driver, be it using @mdev_list or writing the state of a
+ * single ap_matrix_mdev device. It's quite coarse but we don't
+ * expect much contention.
*/
struct ap_matrix_dev {
struct device device;
+ atomic_t available_instances;
+ struct ap_config_info info;
+ struct list_head mdev_list;
+ struct mutex lock;
};
extern struct ap_matrix_dev *matrix_dev;
+/**
+ * The AP matrix is comprised of three bit masks identifying the adapters,
+ * queues (domains) and control domains that belong to an AP matrix. The bits i
+ * each mask, from least significant to most significant bit, correspond to IDs
+ * 0 to 255. When a bit is set, the corresponding ID belongs to the matrix.
+ *
+ * @apm_max: max adapter number in @apm
+ * @apm identifies the AP adapters in the matrix
+ * @aqm_max: max domain number in @aqm
+ * @aqm identifies the AP queues (domains) in the matrix
+ * @adm_max: max domain number in @adm
+ * @adm identifies the AP control domains in the matrix
+ */
+struct ap_matrix {
+ unsigned long apm_max;
+ DECLARE_BITMAP(apm, 256);
+ unsigned long aqm_max;
+ DECLARE_BITMAP(aqm, 256);
+ unsigned long adm_max;
+ DECLARE_BITMAP(adm, 256);
+};
+
+/**
+ * struct ap_matrix_mdev - the mediated matrix device structure
+ * @list: allows the ap_matrix_mdev struct to be added to a list
+ * @matrix: the adapters, usage domains and control domains assigned to the
+ * mediated matrix device.
+ */
+struct ap_matrix_mdev {
+ struct list_head node;
+ struct ap_matrix matrix;
+};
+
+extern int vfio_ap_mdev_register(void);
+extern void vfio_ap_mdev_unregister(void);
+
#endif /* _VFIO_AP_PRIVATE_H_ */
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 1aa7b82..bfbe2be 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -215,6 +215,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
#define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
#define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
+#define VFIO_DEVICE_API_AP_STRING "vfio-ap"
/**
* VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework
2018-09-12 19:42 ` [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
@ 2018-09-20 15:50 ` Cornelia Huck
2018-09-20 20:35 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-20 15:50 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:55 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Registers the matrix device created by the VFIO AP device
> driver with the VFIO mediated device framework.
> Registering the matrix device will create the sysfs
> structures needed to create mediated matrix devices
> each of which will be used to configure the AP matrix
> for a guest and connect it to the VFIO AP device driver.
>
> Registering the matrix device with the VFIO mediated device
> framework will create the following sysfs structures:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ create
>
> To create a mediated device for the AP matrix device, write a UUID
> to the create file:
>
> uuidgen > create
>
> A symbolic link to the mediated device's directory will be created in the
> devices subdirectory named after the generated $uuid:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ [devices]
> ............... [$uuid]
>
> A symbolic link to the mediated device will also be created
> in the vfio_ap matrix's directory:
>
> /sys/devices/vfio_ap/matrix/[$uuid]
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> MAINTAINERS | 1 +
> drivers/s390/crypto/Makefile | 2 +-
> drivers/s390/crypto/vfio_ap_drv.c | 19 +++++
> drivers/s390/crypto/vfio_ap_ops.c | 126 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 49 +++++++++++++
> include/uapi/linux/vfio.h | 1 +
> 6 files changed, 197 insertions(+), 1 deletions(-)
> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
(...)
> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> index 3e635f0..75f8bdc 100644
> --- a/drivers/s390/crypto/vfio_ap_drv.c
> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> @@ -76,6 +76,16 @@ static int vfio_ap_matrix_dev_create(void)
> goto matrix_alloc_err;
> }
>
> + /* Test if PQAP(QCI) instruction is available */
/* Fill in config info via PQAP(QCI), if available */
?
> + if (test_facility(12)) {
> + ret = ap_qci(&matrix_dev->info);
> + if (ret)
> + goto matrix_alloc_err;
> + }
> +
> + mutex_init(&matrix_dev->lock);
> + INIT_LIST_HEAD(&matrix_dev->mdev_list);
> +
> matrix_dev->device.type = &vfio_ap_dev_type;
> dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
> matrix_dev->device.parent = root_device;
(...)
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> index 6141420..a2eab78 100644
> --- a/drivers/s390/crypto/vfio_ap_private.h
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -3,6 +3,7 @@
> * Private data and functions for adjunct processor VFIO matrix driver.
> *
> * Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
> + * Halil Pasic <pasic@linux.ibm.com>
> *
> * Copyright IBM Corp. 2018
> */
> @@ -24,11 +25,59 @@
> /**
> * ap_matrix_dev - the AP matrix device structure
> * @device: generic device structure associated with the AP matrix device
> + * @available_instances: number of mediated matrix devices that can be created
> + * @info: the struct containing the output from the PQAP(TAPQ) instruction
Hm, isn't that rather PQAP(QCI)?
> + * mdev_list: the list of mediated matrix devices created
> + * lock: mutex for locking the AP matrix device. This lock will be
> + * taken every time we fiddle with state managed by the vfio_ap
> + * driver, be it using @mdev_list or writing the state of a
> + * single ap_matrix_mdev device. It's quite coarse but we don't
> + * expect much contention.
> */
> struct ap_matrix_dev {
> struct device device;
> + atomic_t available_instances;
> + struct ap_config_info info;
> + struct list_head mdev_list;
> + struct mutex lock;
> };
Otherwise:
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework
2018-09-20 15:50 ` Cornelia Huck
@ 2018-09-20 20:35 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-20 20:35 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/20/2018 11:50 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:42:55 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Registers the matrix device created by the VFIO AP device
>> driver with the VFIO mediated device framework.
>> Registering the matrix device will create the sysfs
>> structures needed to create mediated matrix devices
>> each of which will be used to configure the AP matrix
>> for a guest and connect it to the VFIO AP device driver.
>>
>> Registering the matrix device with the VFIO mediated device
>> framework will create the following sysfs structures:
>>
>> /sys/devices/vfio_ap/matrix/
>> ...... [mdev_supported_types]
>> ......... [vfio_ap-passthrough]
>> ............ create
>>
>> To create a mediated device for the AP matrix device, write a UUID
>> to the create file:
>>
>> uuidgen > create
>>
>> A symbolic link to the mediated device's directory will be created in the
>> devices subdirectory named after the generated $uuid:
>>
>> /sys/devices/vfio_ap/matrix/
>> ...... [mdev_supported_types]
>> ......... [vfio_ap-passthrough]
>> ............ [devices]
>> ............... [$uuid]
>>
>> A symbolic link to the mediated device will also be created
>> in the vfio_ap matrix's directory:
>>
>> /sys/devices/vfio_ap/matrix/[$uuid]
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> MAINTAINERS | 1 +
>> drivers/s390/crypto/Makefile | 2 +-
>> drivers/s390/crypto/vfio_ap_drv.c | 19 +++++
>> drivers/s390/crypto/vfio_ap_ops.c | 126 +++++++++++++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_private.h | 49 +++++++++++++
>> include/uapi/linux/vfio.h | 1 +
>> 6 files changed, 197 insertions(+), 1 deletions(-)
>> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
> (...)
>
>> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
>> index 3e635f0..75f8bdc 100644
>> --- a/drivers/s390/crypto/vfio_ap_drv.c
>> +++ b/drivers/s390/crypto/vfio_ap_drv.c
>> @@ -76,6 +76,16 @@ static int vfio_ap_matrix_dev_create(void)
>> goto matrix_alloc_err;
>> }
>>
>> + /* Test if PQAP(QCI) instruction is available */
> /* Fill in config info via PQAP(QCI), if available */
Okay
>
> ?
>
>> + if (test_facility(12)) {
>> + ret = ap_qci(&matrix_dev->info);
>> + if (ret)
>> + goto matrix_alloc_err;
>> + }
>> +
>> + mutex_init(&matrix_dev->lock);
>> + INIT_LIST_HEAD(&matrix_dev->mdev_list);
>> +
>> matrix_dev->device.type = &vfio_ap_dev_type;
>> dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
>> matrix_dev->device.parent = root_device;
> (...)
>
>> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
>> index 6141420..a2eab78 100644
>> --- a/drivers/s390/crypto/vfio_ap_private.h
>> +++ b/drivers/s390/crypto/vfio_ap_private.h
>> @@ -3,6 +3,7 @@
>> * Private data and functions for adjunct processor VFIO matrix driver.
>> *
>> * Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
>> + * Halil Pasic <pasic@linux.ibm.com>
>> *
>> * Copyright IBM Corp. 2018
>> */
>> @@ -24,11 +25,59 @@
>> /**
>> * ap_matrix_dev - the AP matrix device structure
>> * @device: generic device structure associated with the AP matrix device
>> + * @available_instances: number of mediated matrix devices that can be created
>> + * @info: the struct containing the output from the PQAP(TAPQ) instruction
> Hm, isn't that rather PQAP(QCI)?
Yes it is!
>
>> + * mdev_list: the list of mediated matrix devices created
>> + * lock: mutex for locking the AP matrix device. This lock will be
>> + * taken every time we fiddle with state managed by the vfio_ap
>> + * driver, be it using @mdev_list or writing the state of a
>> + * single ap_matrix_mdev device. It's quite coarse but we don't
>> + * expect much contention.
>> */
>> struct ap_matrix_dev {
>> struct device device;
>> + atomic_t available_instances;
>> + struct ap_config_info info;
>> + struct list_head mdev_list;
>> + struct mutex lock;
>> };
> Otherwise:
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (4 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 05/26] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-21 9:40 ` Cornelia Huck
2018-09-12 19:42 ` [PATCH v10 07/26] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
` (21 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces two new sysfs attributes for the VFIO mediated
matrix device for assigning AP adapters to and unassigning
AP adapters from a mediated matrix device. The IDs of the
AP adapters assigned to the mediated matrix device will be
stored in an AP mask (APM).
The bits in the APM, from most significant to least significant
bit, correspond to AP adapter IDs (APID) 0 to 255. On
some systems, the maximum allowable adapter number may be less
than 255 - depending upon the host's AP configuration - and
assignment may be rejected if the input adapter ID exceeds the
limit.
When an adapter is assigned, the bit corresponding to the APID
will be set in the APM. Likewise, when an adapter is
unassigned, the bit corresponding to the APID will be cleared
from the APM.
In order to successfully assign an adapter, the APQNs derived from
the adapter ID being assigned and the queue indexes of all domains
previously assigned:
1. Must be bound to the vfio_ap device driver.
2. Must not be assigned to any other mediated matrix device
If there are no domains assigned to the mdev, then there must
be an AP queue bound to the vfio_ap device driver with an
APQN containing the APID, otherwise all domains
subsequently assigned will fail because there will be no
AP queues bound with an APQN containing the adapter ID.
Assigning or un-assigning an AP adapter will be rejected if
a guest using the mediated matrix device is running.
The relevant sysfs structures are:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_adapter
.................. unassign_adapter
To assign an adapter to the $uuid mediated matrix device's APM,
write the APID to the assign_adapter file. To unassign an adapter,
write the APID to the unassign_adapter file. The APID is specified
using conventional semantics: If it begins with 0x the number will
be parsed as a hexadecimal number; if it begins with a 0 the number
will be parsed as an octal number; otherwise, it will be parsed as a
decimal number.
For example, to assign adapter 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_adapter
or
echo 0xad > assign_adapter
or
echo 0255 > assign_adapter
To unassign adapter 173 (0xad):
echo 173 > unassign_adapter
or
echo 0xad > unassign_adapter
or
echo 0255 > unassign_adapter
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 295 +++++++++++++++++++++++++++++++++++++
1 files changed, 295 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 99ed303..1d4f890 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -106,9 +106,304 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
NULL,
};
+struct vfio_ap_queue_reserved {
+ unsigned long *apid;
+ unsigned long *apqi;
+ bool reserved;
+};
+
+/**
+ * vfio_ap_has_queue
+ *
+ * @dev: an AP queue device
+ * @data: a struct vfio_ap_queue_reserved reference
+ *
+ * Flags whether the AP queue device (@dev) has a queue ID containing the APQN,
+ * apid or apqi specified in @data:
+ *
+ * - If @data contains both an apid and apqi value, then @data will be flagged
+ * as reserved if the APID and APQI fields for the AP queue device matches
+ *
+ * - If @data contains only an apid value, @data will be flagged as
+ * reserved if the APID field in the AP queue device matches
+ *
+ * - If @data contains only an apqi value, @data will be flagged as
+ * reserved if the APQI field in the AP queue device matches
+ *
+ * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if
+ * @data does not contain either an apid or apqi.
+ */
+static int vfio_ap_has_queue(struct device *dev, void *data)
+{
+ struct vfio_ap_queue_reserved *qres = data;
+ struct ap_queue *ap_queue = to_ap_queue(dev);
+ ap_qid_t qid;
+ unsigned long id;
+
+ if (qres->apid && qres->apqi) {
+ qid = AP_MKQID(*qres->apid, *qres->apqi);
+ if (qid == ap_queue->qid)
+ qres->reserved = true;
+ } else if (qres->apid && !qres->apqi) {
+ id = AP_QID_CARD(ap_queue->qid);
+ if (id == *qres->apid)
+ qres->reserved = true;
+ } else if (!qres->apid && qres->apqi) {
+ id = AP_QID_QUEUE(ap_queue->qid);
+ if (id == *qres->apqi)
+ qres->reserved = true;
+ } else {
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/**
+ * vfio_ap_verify_queue_reserved
+ *
+ * @matrix_dev: a mediated matrix device
+ * @apid: an AP adapter ID
+ * @apqi: an AP queue index
+ *
+ * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device
+ * driver according to the following rules:
+ *
+ * - If both @apid and @apqi are not NULL, then there must be an AP queue
+ * device bound to the vfio_ap driver with the APQN identified by @apid and
+ * @apqi
+ *
+ * - If only @apid is not NULL, then there must be an AP queue device bound
+ * to the vfio_ap driver with an APQN containing @apid
+ *
+ * - If only @apqi is not NULL, then there must be an AP queue device bound
+ * to the vfio_ap driver with an APQN containing @apqi
+ *
+ * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL.
+ */
+static int vfio_ap_verify_queue_reserved(unsigned long *apid,
+ unsigned long *apqi)
+{
+ int ret;
+ struct vfio_ap_queue_reserved qres;
+
+ qres.apid = apid;
+ qres.apqi = apqi;
+ qres.reserved = false;
+
+ ret = driver_for_each_device(matrix_dev->device.driver, NULL, &qres,
+ vfio_ap_has_queue);
+ if (ret)
+ return ret;
+
+ if (qres.reserved)
+ return 0;
+
+ return -EADDRNOTAVAIL;
+}
+
+static int
+vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ int ret;
+ unsigned long apqi;
+ unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
+
+ if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
+ return vfio_ap_verify_queue_reserved(&apid, NULL);
+
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
+ ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * vfio_ap_mdev_verify_no_sharing
+ *
+ * Verifies that the APQNs derived from the cross product of the AP adapter IDs
+ * and AP queue indexes comprising the AP matrix are not configured for another
+ * mediated device. AP queue sharing is not allowed.
+ *
+ * @kvm: the KVM guest
+ * @matrix: the AP matrix
+ *
+ * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
+ */
+static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+{
+ int nbits;
+ struct ap_matrix_mdev *lstdev;
+ unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)];
+ unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)];
+
+ list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
+ if (matrix_mdev == lstdev)
+ continue;
+
+ memset(apm, 0, sizeof(apm));
+ memset(aqm, 0, sizeof(aqm));
+
+ /*
+ * We work on full longs, as we can only exclude the leftover
+ * bits in non-inverse order. The leftover is all zeros.
+ */
+ nbits = sizeof(apm) * BITS_PER_BYTE;
+ if (!bitmap_and(apm, matrix_mdev->matrix.apm,
+ lstdev->matrix.apm, nbits))
+ continue;
+
+ nbits = sizeof(aqm) * BITS_PER_BYTE;
+ if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
+ lstdev->matrix.aqm, nbits))
+ continue;
+
+ return -EADDRINUSE;
+ }
+
+ return 0;
+}
+
+/**
+ * assign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's assign_adapter attribute
+ * @buf: a buffer containing the AP adapter number (APID) to
+ * be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and sets the corresponding bit in the mediated
+ * matrix device's APM.
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise,
+ * returns one of the following errors:
+ *
+ * 1. -EINVAL
+ * The APID is not a valid number
+ *
+ * 2. -ENODEV
+ * The APID exceeds the maximum value configured for the system
+ *
+ * 3. -EADDRNOTAVAIL
+ * An APQN derived from the cross product of the APID being assigned
+ * and the APQIs previously assigned is not bound to the vfio_ap device
+ * driver; or, if no APQIs have yet been assigned, the APID is not
+ * contained in an APQN bound to the vfio_ap device driver.
+ *
+ * 4. -EADDRINUSE
+ * An APQN derived from the cross product of the APID being assigned
+ * and the APQIs previously assigned is being used by another mediated
+ * matrix device
+ */
+static ssize_t assign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret)
+ return ret;
+
+ if (apid > matrix_mdev->matrix.apm_max)
+ return -ENODEV;
+
+ /* Set the bit in the AP mask (APM) corresponding to the AP adapter
+ * number (APID). The bits in the mask, from most significant to least
+ * significant bit, correspond to APIDs 0-255.
+ */
+ mutex_lock(&matrix_dev->lock);
+
+ ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid);
+ if (ret)
+ goto done;
+
+ set_bit_inv(apid, matrix_mdev->matrix.apm);
+
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ if (ret)
+ goto share_err;
+
+ ret = count;
+ goto done;
+
+share_err:
+ clear_bit_inv(apid, matrix_mdev->matrix.apm);
+done:
+ mutex_unlock(&matrix_dev->lock);
+
+ return ret;
+}
+static DEVICE_ATTR_WO(assign_adapter);
+
+/**
+ * unassign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's unassign_adapter attribute
+ * @buf: a buffer containing the adapter number (APID) to be unassigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and clears the corresponding bit in the mediated
+ * matrix device's APM.
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise,
+ * returns one of the following errors:
+ * -EINVAL if the APID is not a number
+ * -ENODEV if the APID it exceeds the maximum value configured for the
+ * system
+ */
+static ssize_t unassign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret)
+ return ret;
+
+ if (apid > matrix_mdev->matrix.apm_max)
+ return -ENODEV;
+
+ mutex_lock(&matrix_dev->lock);
+ clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
+ mutex_unlock(&matrix_dev->lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_adapter);
+
+static struct attribute *vfio_ap_mdev_attrs[] = {
+ &dev_attr_assign_adapter.attr,
+ &dev_attr_unassign_adapter.attr,
+ NULL
+};
+
+static struct attribute_group vfio_ap_mdev_attr_group = {
+ .attrs = vfio_ap_mdev_attrs
+};
+
+static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
+ &vfio_ap_mdev_attr_group,
+ NULL
+};
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
+ .mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
};
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters
2018-09-12 19:42 ` [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
@ 2018-09-21 9:40 ` Cornelia Huck
2018-09-21 9:52 ` Harald Freudenberger
2018-09-21 14:07 ` Tony Krowiak
0 siblings, 2 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-21 9:40 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:56 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces two new sysfs attributes for the VFIO mediated
> matrix device for assigning AP adapters to and unassigning
> AP adapters from a mediated matrix device. The IDs of the
> AP adapters assigned to the mediated matrix device will be
> stored in an AP mask (APM).
>
> The bits in the APM, from most significant to least significant
> bit, correspond to AP adapter IDs (APID) 0 to 255. On
> some systems, the maximum allowable adapter number may be less
> than 255 - depending upon the host's AP configuration - and
> assignment may be rejected if the input adapter ID exceeds the
> limit.
>
> When an adapter is assigned, the bit corresponding to the APID
> will be set in the APM. Likewise, when an adapter is
> unassigned, the bit corresponding to the APID will be cleared
> from the APM.
>
> In order to successfully assign an adapter, the APQNs derived from
> the adapter ID being assigned and the queue indexes of all domains
> previously assigned:
>
> 1. Must be bound to the vfio_ap device driver.
>
> 2. Must not be assigned to any other mediated matrix device
>
> If there are no domains assigned to the mdev, then there must
> be an AP queue bound to the vfio_ap device driver with an
> APQN containing the APID, otherwise all domains
> subsequently assigned will fail because there will be no
> AP queues bound with an APQN containing the adapter ID.
>
> Assigning or un-assigning an AP adapter will be rejected if
> a guest using the mediated matrix device is running.
>
> The relevant sysfs structures are:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ [devices]
> ...............[$uuid]
> .................. assign_adapter
> .................. unassign_adapter
>
> To assign an adapter to the $uuid mediated matrix device's APM,
> write the APID to the assign_adapter file. To unassign an adapter,
> write the APID to the unassign_adapter file. The APID is specified
> using conventional semantics: If it begins with 0x the number will
> be parsed as a hexadecimal number; if it begins with a 0 the number
> will be parsed as an octal number; otherwise, it will be parsed as a
> decimal number.
>
> For example, to assign adapter 173 (0xad) to the mediated matrix
> device $uuid:
>
> echo 173 > assign_adapter
>
> or
>
> echo 0xad > assign_adapter
>
> or
>
> echo 0255 > assign_adapter
>
> To unassign adapter 173 (0xad):
>
> echo 173 > unassign_adapter
>
> or
>
> echo 0xad > unassign_adapter
>
> or
>
> echo 0255 > unassign_adapter
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 295 +++++++++++++++++++++++++++++++++++++
> 1 files changed, 295 insertions(+), 0 deletions(-)
(...)
> +/**
> + * vfio_ap_mdev_verify_no_sharing
> + *
> + * Verifies that the APQNs derived from the cross product of the AP adapter IDs
> + * and AP queue indexes comprising the AP matrix are not configured for another
> + * mediated device. AP queue sharing is not allowed.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the AP matrix
> + *
> + * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
> + */
> +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
> +{
> + int nbits;
> + struct ap_matrix_mdev *lstdev;
> + unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)];
> + unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)];
Can you please convert this to use a fixed-size array? I think
{apm,aqm}_max has an upper bound of 255?
(Also, this can use DECLARE_BITMAP.)
> +
> + list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
> + if (matrix_mdev == lstdev)
> + continue;
> +
> + memset(apm, 0, sizeof(apm));
> + memset(aqm, 0, sizeof(aqm));
> +
> + /*
> + * We work on full longs, as we can only exclude the leftover
> + * bits in non-inverse order. The leftover is all zeros.
> + */
> + nbits = sizeof(apm) * BITS_PER_BYTE;
> + if (!bitmap_and(apm, matrix_mdev->matrix.apm,
> + lstdev->matrix.apm, nbits))
> + continue;
> +
> + nbits = sizeof(aqm) * BITS_PER_BYTE;
> + if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
> + lstdev->matrix.aqm, nbits))
> + continue;
> +
> + return -EADDRINUSE;
> + }
> +
> + return 0;
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters
2018-09-21 9:40 ` Cornelia Huck
@ 2018-09-21 9:52 ` Harald Freudenberger
2018-09-21 14:07 ` Tony Krowiak
1 sibling, 0 replies; 87+ messages in thread
From: Harald Freudenberger @ 2018-09-21 9:52 UTC (permalink / raw)
To: Tony Krowiak
Cc: Cornelia Huck, linux-s390, linux-kernel, kvm, freude,
schwidefsky, heiko.carstens, borntraeger, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja, Tony Krowiak
On 21.09.2018 11:40, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:42:56 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces two new sysfs attributes for the VFIO mediated
>> matrix device for assigning AP adapters to and unassigning
>> AP adapters from a mediated matrix device. The IDs of the
>> AP adapters assigned to the mediated matrix device will be
>> stored in an AP mask (APM).
>>
>> The bits in the APM, from most significant to least significant
>> bit, correspond to AP adapter IDs (APID) 0 to 255. On
>> some systems, the maximum allowable adapter number may be less
>> than 255 - depending upon the host's AP configuration - and
>> assignment may be rejected if the input adapter ID exceeds the
>> limit.
>>
>> When an adapter is assigned, the bit corresponding to the APID
>> will be set in the APM. Likewise, when an adapter is
>> unassigned, the bit corresponding to the APID will be cleared
>> from the APM.
>>
>> In order to successfully assign an adapter, the APQNs derived from
>> the adapter ID being assigned and the queue indexes of all domains
>> previously assigned:
>>
>> 1. Must be bound to the vfio_ap device driver.
>>
>> 2. Must not be assigned to any other mediated matrix device
>>
>> If there are no domains assigned to the mdev, then there must
>> be an AP queue bound to the vfio_ap device driver with an
>> APQN containing the APID, otherwise all domains
>> subsequently assigned will fail because there will be no
>> AP queues bound with an APQN containing the adapter ID.
>>
>> Assigning or un-assigning an AP adapter will be rejected if
>> a guest using the mediated matrix device is running.
>>
>> The relevant sysfs structures are:
>>
>> /sys/devices/vfio_ap/matrix/
>> ...... [mdev_supported_types]
>> ......... [vfio_ap-passthrough]
>> ............ [devices]
>> ...............[$uuid]
>> .................. assign_adapter
>> .................. unassign_adapter
>>
>> To assign an adapter to the $uuid mediated matrix device's APM,
>> write the APID to the assign_adapter file. To unassign an adapter,
>> write the APID to the unassign_adapter file. The APID is specified
>> using conventional semantics: If it begins with 0x the number will
>> be parsed as a hexadecimal number; if it begins with a 0 the number
>> will be parsed as an octal number; otherwise, it will be parsed as a
>> decimal number.
>>
>> For example, to assign adapter 173 (0xad) to the mediated matrix
>> device $uuid:
>>
>> echo 173 > assign_adapter
>>
>> or
>>
>> echo 0xad > assign_adapter
>>
>> or
>>
>> echo 0255 > assign_adapter
>>
>> To unassign adapter 173 (0xad):
>>
>> echo 173 > unassign_adapter
>>
>> or
>>
>> echo 0xad > unassign_adapter
>>
>> or
>>
>> echo 0255 > unassign_adapter
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> drivers/s390/crypto/vfio_ap_ops.c | 295 +++++++++++++++++++++++++++++++++++++
>> 1 files changed, 295 insertions(+), 0 deletions(-)
> (...)
>
>> +/**
>> + * vfio_ap_mdev_verify_no_sharing
>> + *
>> + * Verifies that the APQNs derived from the cross product of the AP adapter IDs
>> + * and AP queue indexes comprising the AP matrix are not configured for another
>> + * mediated device. AP queue sharing is not allowed.
>> + *
>> + * @kvm: the KVM guest
>> + * @matrix: the AP matrix
>> + *
>> + * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
>> + */
>> +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
>> +{
>> + int nbits;
>> + struct ap_matrix_mdev *lstdev;
>> + unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)];
>> + unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)];
> Can you please convert this to use a fixed-size array? I think
> {apm,aqm}_max has an upper bound of 255?
>
> (Also, this can use DECLARE_BITMAP.)
In ap_bus.h there are upper limit defines for this:
#define AP_DEVICES 256 /* Number of AP devices. */
#define AP_DOMAINS 256 /* Number of AP domains. */
>
>> +
>> + list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
>> + if (matrix_mdev == lstdev)
>> + continue;
>> +
>> + memset(apm, 0, sizeof(apm));
>> + memset(aqm, 0, sizeof(aqm));
>> +
>> + /*
>> + * We work on full longs, as we can only exclude the leftover
>> + * bits in non-inverse order. The leftover is all zeros.
>> + */
>> + nbits = sizeof(apm) * BITS_PER_BYTE;
>> + if (!bitmap_and(apm, matrix_mdev->matrix.apm,
>> + lstdev->matrix.apm, nbits))
>> + continue;
>> +
>> + nbits = sizeof(aqm) * BITS_PER_BYTE;
>> + if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
>> + lstdev->matrix.aqm, nbits))
>> + continue;
>> +
>> + return -EADDRINUSE;
>> + }
>> +
>> + return 0;
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters
2018-09-21 9:40 ` Cornelia Huck
2018-09-21 9:52 ` Harald Freudenberger
@ 2018-09-21 14:07 ` Tony Krowiak
1 sibling, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-21 14:07 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/21/2018 05:40 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:42:56 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces two new sysfs attributes for the VFIO mediated
>> matrix device for assigning AP adapters to and unassigning
>> AP adapters from a mediated matrix device. The IDs of the
>> AP adapters assigned to the mediated matrix device will be
>> stored in an AP mask (APM).
>>
>> The bits in the APM, from most significant to least significant
>> bit, correspond to AP adapter IDs (APID) 0 to 255. On
>> some systems, the maximum allowable adapter number may be less
>> than 255 - depending upon the host's AP configuration - and
>> assignment may be rejected if the input adapter ID exceeds the
>> limit.
>>
>> When an adapter is assigned, the bit corresponding to the APID
>> will be set in the APM. Likewise, when an adapter is
>> unassigned, the bit corresponding to the APID will be cleared
>> from the APM.
>>
>> In order to successfully assign an adapter, the APQNs derived from
>> the adapter ID being assigned and the queue indexes of all domains
>> previously assigned:
>>
>> 1. Must be bound to the vfio_ap device driver.
>>
>> 2. Must not be assigned to any other mediated matrix device
>>
>> If there are no domains assigned to the mdev, then there must
>> be an AP queue bound to the vfio_ap device driver with an
>> APQN containing the APID, otherwise all domains
>> subsequently assigned will fail because there will be no
>> AP queues bound with an APQN containing the adapter ID.
>>
>> Assigning or un-assigning an AP adapter will be rejected if
>> a guest using the mediated matrix device is running.
>>
>> The relevant sysfs structures are:
>>
>> /sys/devices/vfio_ap/matrix/
>> ...... [mdev_supported_types]
>> ......... [vfio_ap-passthrough]
>> ............ [devices]
>> ...............[$uuid]
>> .................. assign_adapter
>> .................. unassign_adapter
>>
>> To assign an adapter to the $uuid mediated matrix device's APM,
>> write the APID to the assign_adapter file. To unassign an adapter,
>> write the APID to the unassign_adapter file. The APID is specified
>> using conventional semantics: If it begins with 0x the number will
>> be parsed as a hexadecimal number; if it begins with a 0 the number
>> will be parsed as an octal number; otherwise, it will be parsed as a
>> decimal number.
>>
>> For example, to assign adapter 173 (0xad) to the mediated matrix
>> device $uuid:
>>
>> echo 173 > assign_adapter
>>
>> or
>>
>> echo 0xad > assign_adapter
>>
>> or
>>
>> echo 0255 > assign_adapter
>>
>> To unassign adapter 173 (0xad):
>>
>> echo 173 > unassign_adapter
>>
>> or
>>
>> echo 0xad > unassign_adapter
>>
>> or
>>
>> echo 0255 > unassign_adapter
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> drivers/s390/crypto/vfio_ap_ops.c | 295 +++++++++++++++++++++++++++++++++++++
>> 1 files changed, 295 insertions(+), 0 deletions(-)
> (...)
>
>> +/**
>> + * vfio_ap_mdev_verify_no_sharing
>> + *
>> + * Verifies that the APQNs derived from the cross product of the AP adapter IDs
>> + * and AP queue indexes comprising the AP matrix are not configured for another
>> + * mediated device. AP queue sharing is not allowed.
>> + *
>> + * @kvm: the KVM guest
>> + * @matrix: the AP matrix
>> + *
>> + * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
>> + */
>> +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
>> +{
>> + int nbits;
>> + struct ap_matrix_mdev *lstdev;
>> + unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)];
>> + unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)];
> Can you please convert this to use a fixed-size array? I think
> {apm,aqm}_max has an upper bound of 255?
Sure
>
> (Also, this can use DECLARE_BITMAP.)
Indeed it can.
>
>> +
>> + list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
>> + if (matrix_mdev == lstdev)
>> + continue;
>> +
>> + memset(apm, 0, sizeof(apm));
>> + memset(aqm, 0, sizeof(aqm));
>> +
>> + /*
>> + * We work on full longs, as we can only exclude the leftover
>> + * bits in non-inverse order. The leftover is all zeros.
>> + */
>> + nbits = sizeof(apm) * BITS_PER_BYTE;
>> + if (!bitmap_and(apm, matrix_mdev->matrix.apm,
>> + lstdev->matrix.apm, nbits))
>> + continue;
>> +
>> + nbits = sizeof(aqm) * BITS_PER_BYTE;
>> + if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
>> + lstdev->matrix.aqm, nbits))
>> + continue;
>> +
>> + return -EADDRINUSE;
>> + }
>> +
>> + return 0;
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 07/26] s390: vfio-ap: sysfs interfaces to configure domains
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (5 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 06/26] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-24 10:45 ` Cornelia Huck
2018-09-12 19:42 ` [PATCH v10 08/26] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
` (20 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces two new sysfs attributes for the VFIO mediated
matrix device for assigning AP domains to and unassigning
AP domains from a mediated matrix device. The IDs of the
AP domains assigned to the mediated matrix device will be
stored in an AP queue mask (AQM).
The bits in the AQM, from most significant to least
significant bit, correspond to AP queue index (APQI) 0 to
255 (note that an APQI is synonymous with with a domain ID).
On some systems, the maximum allowable domain number may be
less than 255 - depending upon the host's AP configuration -
and assignment may be rejected if the input domain ID exceeds
the limit.
When a domain is assigned, the bit corresponding to the APQI
will be set in the AQM. Likewise, when a domain is unassigned,
the bit corresponding to the APQI will be cleared from the AQM.
In order to successfully assign a domain, the APQNs derived from
the domain ID being assigned and the adapter numbers of all
adapters previously assigned:
1. Must be bound to the vfio_ap device driver.
2. Must not be assigned to any other mediated matrix device.
If there are no adapters assigned to the mdev, then there must
be an AP queue bound to the vfio_ap device driver with an
APQN containing the domain ID (i.e., APQI), otherwise all
adapters subsequently assigned will fail because there will be no
AP queues bound with an APQN containing the APQI.
Assigning or un-assigning an AP domain will also be rejected if
a guest using the mediated matrix device is running.
The relevant sysfs structures are:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_domain
.................. unassign_domain
To assign a domain to the $uuid mediated matrix device,
write the domain's ID to the assign_domain file. To
unassign a domain, write the domain's ID to the
unassign_domain file. The ID is specified using
conventional semantics: If it begins with 0x, the number
will be parsed as a hexadecimal (case insensitive) number;
if it begins with 0, it will be parsed as an octal number;
otherwise, it will be parsed as a decimal number.
For example, to assign domain 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_domain
or
echo 0255 > assign_domain
or
echo 0xad > assign_domain
To unassign domain 173 (0xad):
echo 173 > unassign_domain
or
echo 0255 > unassign_domain
or
echo 0xad > unassign_domain
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 138 ++++++++++++++++++++++++++++++++++++-
1 files changed, 137 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 1d4f890..a687624 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -385,10 +385,146 @@ static ssize_t unassign_adapter_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_adapter);
+static int
+vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ int ret;
+ unsigned long apid;
+ unsigned long nbits = matrix_mdev->matrix.apm_max + 1;
+
+ if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits)
+ return vfio_ap_verify_queue_reserved(NULL, &apqi);
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) {
+ ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * assign_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's assign_domain attribute
+ * @buf: a buffer containing the AP queue index (APQI) of the domain to
+ * be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APQI from @buf and sets the corresponding bit in the mediated
+ * matrix device's AQM.
+ *
+ * Returns the number of bytes processed if the APQI is valid; otherwise returns
+ * one of the following errors:
+ *
+ * 1. -EINVAL
+ * The APQI is not a valid number
+ *
+ * 2. -ENODEV
+ * The APQI exceeds the maximum value configured for the system
+ *
+ * 3. -EADDRNOTAVAIL
+ * An APQN derived from the cross product of the APQI being assigned
+ * and the APIDs previously assigned is not bound to the vfio_ap device
+ * driver; or, if no APIDs have yet been assigned, the APQI is not
+ * contained in an APQN bound to the vfio_ap device driver.
+ *
+ * 4. -EADDRINUSE
+ * An APQN derived from the cross product of the APQI being assigned
+ * and the APIDs previously assigned is being used by another mediated
+ * matrix device
+ */
+static ssize_t assign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret)
+ return ret;
+ if (apqi > max_apqi)
+ return -ENODEV;
+
+ mutex_lock(&matrix_dev->lock);
+
+ ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi);
+ if (ret)
+ goto done;
+
+ set_bit_inv(apqi, matrix_mdev->matrix.aqm);
+
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ if (ret)
+ goto share_err;
+
+ ret = count;
+ goto done;
+
+share_err:
+ clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
+done:
+ mutex_unlock(&matrix_dev->lock);
+
+ return ret;
+}
+DEVICE_ATTR_WO(assign_domain);
+
+
+/**
+ * unassign_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's unassign_domain attribute
+ * @buf: a buffer containing the AP queue index (APQI) of the domain to
+ * be unassigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APQI from @buf and clears the corresponding bit in the
+ * mediated matrix device's AQM.
+ *
+ * Returns the number of bytes processed if the APQI is valid; otherwise,
+ * returns one of the following errors:
+ * -EINVAL if the APQI is not a number
+ * -ENODEV if the APQI exceeds the maximum value configured for the system
+ */
+static ssize_t unassign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret)
+ return ret;
+
+ if (apqi > matrix_mdev->matrix.aqm_max)
+ return -ENODEV;
+
+ mutex_lock(&matrix_dev->lock);
+ clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
+ mutex_unlock(&matrix_dev->lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_domain);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
- NULL
+ &dev_attr_assign_domain.attr,
+ &dev_attr_unassign_domain.attr,
+ NULL,
};
static struct attribute_group vfio_ap_mdev_attr_group = {
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 07/26] s390: vfio-ap: sysfs interfaces to configure domains
2018-09-12 19:42 ` [PATCH v10 07/26] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
@ 2018-09-24 10:45 ` Cornelia Huck
0 siblings, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 10:45 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:57 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces two new sysfs attributes for the VFIO mediated
> matrix device for assigning AP domains to and unassigning
> AP domains from a mediated matrix device. The IDs of the
> AP domains assigned to the mediated matrix device will be
> stored in an AP queue mask (AQM).
>
> The bits in the AQM, from most significant to least
> significant bit, correspond to AP queue index (APQI) 0 to
> 255 (note that an APQI is synonymous with with a domain ID).
> On some systems, the maximum allowable domain number may be
> less than 255 - depending upon the host's AP configuration -
> and assignment may be rejected if the input domain ID exceeds
> the limit.
>
> When a domain is assigned, the bit corresponding to the APQI
> will be set in the AQM. Likewise, when a domain is unassigned,
> the bit corresponding to the APQI will be cleared from the AQM.
>
> In order to successfully assign a domain, the APQNs derived from
> the domain ID being assigned and the adapter numbers of all
> adapters previously assigned:
>
> 1. Must be bound to the vfio_ap device driver.
>
> 2. Must not be assigned to any other mediated matrix device.
>
> If there are no adapters assigned to the mdev, then there must
> be an AP queue bound to the vfio_ap device driver with an
> APQN containing the domain ID (i.e., APQI), otherwise all
> adapters subsequently assigned will fail because there will be no
> AP queues bound with an APQN containing the APQI.
>
> Assigning or un-assigning an AP domain will also be rejected if
> a guest using the mediated matrix device is running.
>
> The relevant sysfs structures are:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ [devices]
> ...............[$uuid]
> .................. assign_domain
> .................. unassign_domain
>
> To assign a domain to the $uuid mediated matrix device,
> write the domain's ID to the assign_domain file. To
> unassign a domain, write the domain's ID to the
> unassign_domain file. The ID is specified using
> conventional semantics: If it begins with 0x, the number
> will be parsed as a hexadecimal (case insensitive) number;
> if it begins with 0, it will be parsed as an octal number;
> otherwise, it will be parsed as a decimal number.
>
> For example, to assign domain 173 (0xad) to the mediated matrix
> device $uuid:
>
> echo 173 > assign_domain
>
> or
>
> echo 0255 > assign_domain
>
> or
>
> echo 0xad > assign_domain
>
> To unassign domain 173 (0xad):
>
> echo 173 > unassign_domain
>
> or
>
> echo 0255 > unassign_domain
>
> or
>
> echo 0xad > unassign_domain
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 138 ++++++++++++++++++++++++++++++++++++-
> 1 files changed, 137 insertions(+), 1 deletions(-)
Might have been good to add some documentation for the new attributes
in the sysfs API directory, but as that one seems to be rather
hit-and-miss anyway (and you provide a documentation file for the whole
thing),
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 08/26] s390: vfio-ap: sysfs interfaces to configure control domains
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (6 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 07/26] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-24 10:57 ` Cornelia Huck
2018-09-12 19:42 ` [PATCH v10 09/26] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
` (19 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Provides the sysfs interfaces for:
1. Assigning AP control domains to the mediated matrix device
2. Unassigning AP control domains from a mediated matrix device
3. Displaying the control domains assigned to a mediated matrix
device
The IDs of the AP control domains assigned to the mediated matrix
device are stored in an AP domain mask (ADM). The bits in the ADM,
from most significant to least significant bit, correspond to
AP domain numbers 0 to 255. On some systems, the maximum allowable
domain number may be less than 255 - depending upon the host's
AP configuration - and assignment may be rejected if the input
domain ID exceeds the limit.
When a control domain is assigned, the bit corresponding its domain
ID will be set in the ADM. Likewise, when a domain is unassigned,
the bit corresponding to its domain ID will be cleared in the ADM.
The relevant sysfs structures are:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_control_domain
.................. unassign_control_domain
To assign a control domain to the $uuid mediated matrix device's
ADM, write its domain number to the assign_control_domain file.
To unassign a domain, write its domain number to the
unassign_control_domain file. The domain number is specified
using conventional semantics: If it begins with 0x the number
will be parsed as a hexadecimal (case insensitive) number;
if it begins with 0, it is parsed as an octal number;
otherwise, it will be parsed as a decimal number.
For example, to assign control domain 173 (0xad) to the mediated
matrix device $uuid:
echo 173 > assign_control_domain
or
echo 0255 > assign_control_domain
or
echo 0xad > assign_control_domain
To unassign control domain 173 (0xad):
echo 173 > unassign_control_domain
or
echo 0255 > unassign_control_domain
or
echo 0xad > unassign_control_domain
The assignment will be rejected if the APQI exceeds the maximum
value for an AP domain:
* If the AP Extended Addressing (APXA) facility is installed,
the max value is 255
* Else the max value is 15
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 112 +++++++++++++++++++++++++++++++++++++
1 files changed, 112 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index a687624..179b093 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -519,11 +519,123 @@ static ssize_t unassign_domain_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_domain);
+/**
+ * assign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's assign_control_domain attribute
+ * @buf: a buffer containing the domain ID to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and sets the corresponding bit in the mediated
+ * matrix device's ADM.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise,
+ * returns one of the following errors:
+ * -EINVAL if the ID is not a number
+ * -ENODEV if the ID exceeds the maximum value configured for the system
+ */
+static ssize_t assign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long id;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &id);
+ if (ret)
+ return ret;
+
+ if (id > matrix_mdev->matrix.adm_max)
+ return -ENODEV;
+
+ /* Set the bit in the ADM (bitmask) corresponding to the AP control
+ * domain number (id). The bits in the mask, from most significant to
+ * least significant, correspond to IDs 0 up to the one less than the
+ * number of control domains that can be assigned.
+ */
+ mutex_lock(&matrix_dev->lock);
+ set_bit_inv(id, matrix_mdev->matrix.adm);
+ mutex_unlock(&matrix_dev->lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(assign_control_domain);
+
+/**
+ * unassign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's unassign_control_domain attribute
+ * @buf: a buffer containing the domain ID to be unassigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and clears the corresponding bit in the
+ * mediated matrix device's ADM.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise,
+ * returns one of the following errors:
+ * -EINVAL if the ID is not a number
+ * -ENODEV if the ID exceeds the maximum value configured for the system
+ */
+static ssize_t unassign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long domid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_domid = matrix_mdev->matrix.adm_max;
+
+ ret = kstrtoul(buf, 0, &domid);
+ if (ret)
+ return ret;
+ if (domid > max_domid)
+ return -ENODEV;
+
+ mutex_lock(&matrix_dev->lock);
+ clear_bit_inv(domid, matrix_mdev->matrix.adm);
+ mutex_unlock(&matrix_dev->lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_control_domain);
+
+static ssize_t control_domains_show(struct device *dev,
+ struct device_attribute *dev_attr,
+ char *buf)
+{
+ unsigned long id;
+ int nchars = 0;
+ int n;
+ char *bufpos = buf;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_domid = matrix_mdev->matrix.adm_max;
+
+ mutex_lock(&matrix_dev->lock);
+ for_each_set_bit_inv(id, matrix_mdev->matrix.adm, max_domid + 1) {
+ n = sprintf(bufpos, "%04lx\n", id);
+ bufpos += n;
+ nchars += n;
+ }
+ mutex_unlock(&matrix_dev->lock);
+
+ return nchars;
+}
+DEVICE_ATTR_RO(control_domains);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
&dev_attr_assign_domain.attr,
&dev_attr_unassign_domain.attr,
+ &dev_attr_assign_control_domain.attr,
+ &dev_attr_unassign_control_domain.attr,
+ &dev_attr_control_domains.attr,
NULL,
};
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 08/26] s390: vfio-ap: sysfs interfaces to configure control domains
2018-09-12 19:42 ` [PATCH v10 08/26] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
@ 2018-09-24 10:57 ` Cornelia Huck
0 siblings, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 10:57 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:58 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Provides the sysfs interfaces for:
>
> 1. Assigning AP control domains to the mediated matrix device
>
> 2. Unassigning AP control domains from a mediated matrix device
>
> 3. Displaying the control domains assigned to a mediated matrix
> device
>
> The IDs of the AP control domains assigned to the mediated matrix
> device are stored in an AP domain mask (ADM). The bits in the ADM,
> from most significant to least significant bit, correspond to
> AP domain numbers 0 to 255. On some systems, the maximum allowable
> domain number may be less than 255 - depending upon the host's
> AP configuration - and assignment may be rejected if the input
> domain ID exceeds the limit.
>
> When a control domain is assigned, the bit corresponding its domain
> ID will be set in the ADM. Likewise, when a domain is unassigned,
> the bit corresponding to its domain ID will be cleared in the ADM.
>
> The relevant sysfs structures are:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ [devices]
> ...............[$uuid]
> .................. assign_control_domain
> .................. unassign_control_domain
>
> To assign a control domain to the $uuid mediated matrix device's
> ADM, write its domain number to the assign_control_domain file.
> To unassign a domain, write its domain number to the
> unassign_control_domain file. The domain number is specified
> using conventional semantics: If it begins with 0x the number
> will be parsed as a hexadecimal (case insensitive) number;
> if it begins with 0, it is parsed as an octal number;
> otherwise, it will be parsed as a decimal number.
>
> For example, to assign control domain 173 (0xad) to the mediated
> matrix device $uuid:
>
> echo 173 > assign_control_domain
>
> or
>
> echo 0255 > assign_control_domain
>
> or
>
> echo 0xad > assign_control_domain
>
> To unassign control domain 173 (0xad):
>
> echo 173 > unassign_control_domain
>
> or
>
> echo 0255 > unassign_control_domain
>
> or
>
> echo 0xad > unassign_control_domain
>
> The assignment will be rejected if the APQI exceeds the maximum
> value for an AP domain:
> * If the AP Extended Addressing (APXA) facility is installed,
> the max value is 255
> * Else the max value is 15
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 112 +++++++++++++++++++++++++++++++++++++
> 1 files changed, 112 insertions(+), 0 deletions(-)
>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 09/26] s390: vfio-ap: sysfs interface to view matrix mdev matrix
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (7 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 08/26] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
@ 2018-09-12 19:42 ` Tony Krowiak
2018-09-24 10:59 ` Cornelia Huck
2018-09-12 19:43 ` [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks Tony Krowiak
` (18 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:42 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Provides a sysfs interface to view the AP matrix configured for the
mediated matrix device.
The relevant sysfs structures are:
/sys/devices/vfio_ap/matrix/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. matrix
To view the matrix configured for the mediated matrix device,
print the matrix file:
cat matrix
Below are examples of the output from the above command:
Example 1: Adapters and domains assigned
Assignments:
Adapters 5 and 6
Domains 4 and 71 (0x47)
Output
05.0004
05.0047
06.0004
06.0047
Examples 2: Only adapters assigned
Assignments:
Adapters 5 and 6
Output:
05.
06.
Examples 3: Only domains assigned
Assignments:
Domains 4 and 71 (0x47)
Output:
.0004
.0047
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 51 +++++++++++++++++++++++++++++++++++++
1 files changed, 51 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 179b093..2a2dcf7 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -628,6 +628,56 @@ static ssize_t control_domains_show(struct device *dev,
}
DEVICE_ATTR_RO(control_domains);
+static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ char *bufpos = buf;
+ unsigned long apid;
+ unsigned long apqi;
+ unsigned long apid1;
+ unsigned long apqi1;
+ unsigned long napm_bits = matrix_mdev->matrix.apm_max + 1;
+ unsigned long naqm_bits = matrix_mdev->matrix.aqm_max + 1;
+ int nchars = 0;
+ int n;
+
+ apid1 = find_first_bit_inv(matrix_mdev->matrix.apm, napm_bits);
+ apqi1 = find_first_bit_inv(matrix_mdev->matrix.aqm, naqm_bits);
+
+ mutex_lock(&matrix_dev->lock);
+
+ if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) {
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+ naqm_bits) {
+ n = sprintf(bufpos, "%02lx.%04lx\n", apid,
+ apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+ } else if (apid1 < napm_bits) {
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
+ n = sprintf(bufpos, "%02lx.\n", apid);
+ bufpos += n;
+ nchars += n;
+ }
+ } else if (apqi1 < naqm_bits) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, naqm_bits) {
+ n = sprintf(bufpos, ".%04lx\n", apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+
+ mutex_unlock(&matrix_dev->lock);
+
+ return nchars;
+}
+DEVICE_ATTR_RO(matrix);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
@@ -636,6 +686,7 @@ static ssize_t control_domains_show(struct device *dev,
&dev_attr_assign_control_domain.attr,
&dev_attr_unassign_control_domain.attr,
&dev_attr_control_domains.attr,
+ &dev_attr_matrix.attr,
NULL,
};
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 09/26] s390: vfio-ap: sysfs interface to view matrix mdev matrix
2018-09-12 19:42 ` [PATCH v10 09/26] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
@ 2018-09-24 10:59 ` Cornelia Huck
0 siblings, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 10:59 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:59 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Provides a sysfs interface to view the AP matrix configured for the
> mediated matrix device.
>
> The relevant sysfs structures are:
>
> /sys/devices/vfio_ap/matrix/
> ...... [mdev_supported_types]
> ......... [vfio_ap-passthrough]
> ............ [devices]
> ...............[$uuid]
> .................. matrix
>
> To view the matrix configured for the mediated matrix device,
> print the matrix file:
>
> cat matrix
>
> Below are examples of the output from the above command:
>
> Example 1: Adapters and domains assigned
> Assignments:
> Adapters 5 and 6
> Domains 4 and 71 (0x47)
>
> Output
> 05.0004
> 05.0047
> 06.0004
> 06.0047
>
> Examples 2: Only adapters assigned
> Assignments:
> Adapters 5 and 6
>
> Output:
> 05.
> 06.
>
> Examples 3: Only domains assigned
> Assignments:
> Domains 4 and 71 (0x47)
>
> Output:
> .0004
> .0047
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 51 +++++++++++++++++++++++++++++++++++++
> 1 files changed, 51 insertions(+), 0 deletions(-)
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (8 preceding siblings ...)
2018-09-12 19:42 ` [PATCH v10 09/26] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-24 11:01 ` Cornelia Huck
2018-09-12 19:43 ` [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback Tony Krowiak
` (17 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces two new KVM interface to clear the APM, AQM and ADM masks in
the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
not get out of sync.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Acked-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
arch/s390/include/asm/kvm_host.h | 2 ++
arch/s390/kvm/kvm-s390.c | 15 +++++++++++++++
2 files changed, 17 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 423cce7..1e758fe 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -858,6 +858,8 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
struct kvm_async_pf *work);
+void kvm_arch_crypto_clear_masks(struct kvm *kvm);
+
extern int sie64a(struct kvm_s390_sie_block *, u64 *);
extern char sie_exit;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index d717041..ac4c93f 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2033,6 +2033,21 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
}
+void kvm_arch_crypto_clear_masks(struct kvm *kvm)
+{
+ mutex_lock(&kvm->lock);
+ kvm_s390_vcpu_block_all(kvm);
+
+ memset(&kvm->arch.crypto.crycb->apcb0, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb0));
+ memset(&kvm->arch.crypto.crycb->apcb1, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb1));
+
+ kvm_s390_vcpu_unblock_all(kvm);
+ mutex_unlock(&kvm->lock);
+}
+EXPORT_SYMBOL_GPL(kvm_arch_crypto_clear_masks);
+
static u64 kvm_s390_get_initial_cpuid(void)
{
struct cpuid cpuid;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-12 19:43 ` [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks Tony Krowiak
@ 2018-09-24 11:01 ` Cornelia Huck
2018-09-24 11:50 ` Halil Pasic
2018-09-24 14:49 ` Tony Krowiak
0 siblings, 2 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:01 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:43:00 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces two new KVM interface to clear the APM, AQM and ADM masks in
> the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
> not get out of sync.
Hm, that patch description does not quite match what the patch actually
does...
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Acked-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/include/asm/kvm_host.h | 2 ++
> arch/s390/kvm/kvm-s390.c | 15 +++++++++++++++
> 2 files changed, 17 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 423cce7..1e758fe 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -858,6 +858,8 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
> void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
> struct kvm_async_pf *work);
>
> +void kvm_arch_crypto_clear_masks(struct kvm *kvm);
> +
> extern int sie64a(struct kvm_s390_sie_block *, u64 *);
> extern char sie_exit;
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index d717041..ac4c93f 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -2033,6 +2033,21 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
> kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
> }
>
> +void kvm_arch_crypto_clear_masks(struct kvm *kvm)
> +{
> + mutex_lock(&kvm->lock);
> + kvm_s390_vcpu_block_all(kvm);
> +
> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb0));
> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb1));
> +
> + kvm_s390_vcpu_unblock_all(kvm);
> + mutex_unlock(&kvm->lock);
> +}
> +EXPORT_SYMBOL_GPL(kvm_arch_crypto_clear_masks);
...although this function looks fine.
> +
> static u64 kvm_s390_get_initial_cpuid(void)
> {
> struct cpuid cpuid;
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-24 11:01 ` Cornelia Huck
@ 2018-09-24 11:50 ` Halil Pasic
2018-09-24 12:01 ` Cornelia Huck
2018-09-24 14:49 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: Halil Pasic @ 2018-09-24 11:50 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On 09/24/2018 01:01 PM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:43:00 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces two new KVM interface to clear the APM, AQM and ADM masks in
>> the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
>> not get out of sync.
>
> Hm, that patch description does not quite match what the patch actually
> does...
>
You mean it is not obvious where the masks get cleared? Well the APCB is
defined to contain exactly the three masks. As far as I can tell the patch
description is correct.
Regards,
Halil
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Acked-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> arch/s390/include/asm/kvm_host.h | 2 ++
>> arch/s390/kvm/kvm-s390.c | 15 +++++++++++++++
>> 2 files changed, 17 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 423cce7..1e758fe 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -858,6 +858,8 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
>> void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
>> struct kvm_async_pf *work);
>>
>> +void kvm_arch_crypto_clear_masks(struct kvm *kvm);
>> +
>> extern int sie64a(struct kvm_s390_sie_block *, u64 *);
>> extern char sie_exit;
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index d717041..ac4c93f 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -2033,6 +2033,21 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>> kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
>> }
>>
>> +void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> +{
>> + mutex_lock(&kvm->lock);
>> + kvm_s390_vcpu_block_all(kvm);
>> +
>> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
>> + sizeof(kvm->arch.crypto.crycb->apcb0));
>> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
>> + sizeof(kvm->arch.crypto.crycb->apcb1));
>> +
>> + kvm_s390_vcpu_unblock_all(kvm);
>> + mutex_unlock(&kvm->lock);
>> +}
>> +EXPORT_SYMBOL_GPL(kvm_arch_crypto_clear_masks);
>
> ...although this function looks fine.
>
>> +
>> static u64 kvm_s390_get_initial_cpuid(void)
>> {
>> struct cpuid cpuid;
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-24 11:50 ` Halil Pasic
@ 2018-09-24 12:01 ` Cornelia Huck
2018-09-24 15:33 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 12:01 UTC (permalink / raw)
To: Halil Pasic
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Mon, 24 Sep 2018 13:50:36 +0200
Halil Pasic <pasic@linux.ibm.com> wrote:
> On 09/24/2018 01:01 PM, Cornelia Huck wrote:
> > On Wed, 12 Sep 2018 15:43:00 -0400
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >
> >> From: Tony Krowiak <akrowiak@linux.ibm.com>
> >>
> >> Introduces two new KVM interface to clear the APM, AQM and ADM masks in
> >> the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
> >> not get out of sync.
> >
> > Hm, that patch description does not quite match what the patch actually
> > does...
> >
>
> You mean it is not obvious where the masks get cleared? Well the APCB is
> defined to contain exactly the three masks. As far as I can tell the patch
> description is correct.
No, I was referring to "two new KVM interface[s]". The patch just
introduces a new function that clears the masks (well, maybe you can
call this an "interface"). So
s/two new KVM interface/a function in KVM/
and I'd be happy :) (sorry for being too vague)
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-24 12:01 ` Cornelia Huck
@ 2018-09-24 15:33 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 15:33 UTC (permalink / raw)
To: Cornelia Huck, Halil Pasic
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/24/2018 08:01 AM, Cornelia Huck wrote:
> On Mon, 24 Sep 2018 13:50:36 +0200
> Halil Pasic <pasic@linux.ibm.com> wrote:
>
>> On 09/24/2018 01:01 PM, Cornelia Huck wrote:
>>> On Wed, 12 Sep 2018 15:43:00 -0400
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>
>>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>
>>>> Introduces two new KVM interface to clear the APM, AQM and ADM masks in
>>>> the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
>>>> not get out of sync.
>>>
>>> Hm, that patch description does not quite match what the patch actually
>>> does...
>>>
>>
>> You mean it is not obvious where the masks get cleared? Well the APCB is
>> defined to contain exactly the three masks. As far as I can tell the patch
>> description is correct.
>
> No, I was referring to "two new KVM interface[s]". The patch just
> introduces a new function that clears the masks (well, maybe you can
> call this an "interface"). So
>
> s/two new KVM interface/a function in KVM/
>
> and I'd be happy :) (sorry for being too vague)
I'm not quite sure why this is not an interface, but this is not a hill
I'm willing to die on. And, I agree, there are not two of them. I'll
make the change.
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks
2018-09-24 11:01 ` Cornelia Huck
2018-09-24 11:50 ` Halil Pasic
@ 2018-09-24 14:49 ` Tony Krowiak
1 sibling, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 14:49 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/24/2018 07:01 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:43:00 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces two new KVM interface to clear the APM, AQM and ADM masks in
>> the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
>> not get out of sync.
>
> Hm, that patch description does not quite match what the patch actually
> does...
No, it doesn't, does it? I'll fix it.
>
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Acked-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> arch/s390/include/asm/kvm_host.h | 2 ++
>> arch/s390/kvm/kvm-s390.c | 15 +++++++++++++++
>> 2 files changed, 17 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 423cce7..1e758fe 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -858,6 +858,8 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
>> void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
>> struct kvm_async_pf *work);
>>
>> +void kvm_arch_crypto_clear_masks(struct kvm *kvm);
>> +
>> extern int sie64a(struct kvm_s390_sie_block *, u64 *);
>> extern char sie_exit;
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index d717041..ac4c93f 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -2033,6 +2033,21 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>> kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
>> }
>>
>> +void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> +{
>> + mutex_lock(&kvm->lock);
>> + kvm_s390_vcpu_block_all(kvm);
>> +
>> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
>> + sizeof(kvm->arch.crypto.crycb->apcb0));
>> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
>> + sizeof(kvm->arch.crypto.crycb->apcb1));
>> +
>> + kvm_s390_vcpu_unblock_all(kvm);
>> + mutex_unlock(&kvm->lock);
>> +}
>> +EXPORT_SYMBOL_GPL(kvm_arch_crypto_clear_masks);
>
> ...although this function looks fine.
>
>> +
>> static u64 kvm_s390_get_initial_cpuid(void)
>> {
>> struct cpuid cpuid;
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (9 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 10/26] KVM: s390: interfaces to clear CRYCB masks Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-18 17:00 ` Halil Pasic
2018-09-21 23:28 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
` (16 subsequent siblings)
27 siblings, 2 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Implements the open callback on the mediated matrix device.
The function registers a group notifier to receive notification
of the VFIO_GROUP_NOTIFY_SET_KVM event. When notified,
the vfio_ap device driver will get access to the guest's
kvm structure. The open callback must ensure that only one
mediated device shall be opened per guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Acked-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
arch/s390/include/asm/kvm_host.h | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 168 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 5 +
3 files changed, 174 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 1e758fe..b32bd1b 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -258,6 +258,7 @@ struct kvm_s390_sie_block {
__u64 tecmc; /* 0x00e8 */
__u8 reservedf0[12]; /* 0x00f0 */
#define CRYCB_FORMAT_MASK 0x00000003
+#define CRYCB_FORMAT0 0x00000000
#define CRYCB_FORMAT1 0x00000001
#define CRYCB_FORMAT2 0x00000003
__u32 crycbd; /* 0x00fc */
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 2a2dcf7..8bc0cdd 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -13,6 +13,10 @@
#include <linux/device.h>
#include <linux/list.h>
#include <linux/ctype.h>
+#include <linux/bitops.h>
+#include <linux/kvm_host.h>
+#include <linux/module.h>
+#include <asm/kvm.h>
#include <asm/zcrypt.h>
#include "vfio_ap_private.h"
@@ -54,6 +58,9 @@ static int vfio_ap_mdev_remove(struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
mutex_lock(&matrix_dev->lock);
list_del(&matrix_mdev->node);
mutex_unlock(&matrix_dev->lock);
@@ -309,6 +316,10 @@ static ssize_t assign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow assignment of adapter */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -370,6 +381,10 @@ static ssize_t unassign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow un-assignment of adapter */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -447,6 +462,10 @@ static ssize_t assign_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
+ /* If the guest is running, disallow assignment of domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -504,6 +523,10 @@ static ssize_t unassign_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow un-assignment of domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -544,6 +567,10 @@ static ssize_t assign_control_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow assignment of control domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &id);
if (ret)
return ret;
@@ -590,6 +617,10 @@ static ssize_t unassign_control_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_domid = matrix_mdev->matrix.adm_max;
+ /* If the guest is running, disallow un-assignment of control domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &domid);
if (ret)
return ret;
@@ -699,12 +730,149 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
NULL
};
+/**
+ * Verify that the AP instructions are available on the guest. This is indicated
+ * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
+ */
+static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
+{
+ if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
+ return 0;
+
+ return -EOPNOTSUPP;
+}
+
+static void vfio_ap_mdev_copy_masks(struct ap_matrix_mdev *matrix_mdev)
+{
+ int nbytes;
+ unsigned long *apm, *aqm, *adm;
+ struct kvm_s390_crypto_cb *crycb = matrix_mdev->kvm->arch.crypto.crycb;
+
+ switch (matrix_mdev->kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK) {
+ case CRYCB_FORMAT2:
+ apm = (unsigned long *)crycb->apcb1.apm;
+ aqm = (unsigned long *)crycb->apcb1.aqm;
+ adm = (unsigned long *)crycb->apcb1.adm;
+ break;
+ case CRYCB_FORMAT1:
+ case CRYCB_FORMAT0:
+ default:
+ apm = (unsigned long *)crycb->apcb0.apm;
+ aqm = (unsigned long *)crycb->apcb0.aqm;
+ adm = (unsigned long *)crycb->apcb0.adm;
+ break;
+ }
+
+ nbytes = DIV_ROUND_UP(matrix_mdev->matrix.apm_max + 1, BITS_PER_BYTE);
+ memcpy(apm, matrix_mdev->matrix.apm, nbytes);
+ nbytes = DIV_ROUND_UP(matrix_mdev->matrix.aqm_max + 1, BITS_PER_BYTE);
+ memcpy(aqm, matrix_mdev->matrix.aqm, nbytes);
+ nbytes = DIV_ROUND_UP(matrix_mdev->matrix.adm_max + 1, BITS_PER_BYTE);
+ memcpy(adm, matrix_mdev->matrix.adm, nbytes);
+}
+
+/**
+ * vfio_ap_mdev_open_once
+ *
+ * @matrix_mdev: a mediated matrix device
+ *
+ * Return 0 if no other mediated matrix device has been opened for the
+ * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
+ */
+static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
+ struct kvm *kvm)
+{
+ struct ap_matrix_mdev *m;
+
+ mutex_lock(&matrix_dev->lock);
+
+ list_for_each_entry(m, &matrix_dev->mdev_list, node) {
+ if ((m != matrix_mdev) && (m->kvm == kvm)) {
+ mutex_unlock(&matrix_dev->lock);
+ return -EPERM;
+ }
+ }
+
+ mutex_unlock(&matrix_dev->lock);
+
+ return 0;
+}
+
+static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ int ret;
+ struct ap_matrix_mdev *matrix_mdev;
+
+ if (action != VFIO_GROUP_NOTIFY_SET_KVM)
+ return NOTIFY_OK;
+
+ matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
+
+ if (!data) {
+ matrix_mdev->kvm = NULL;
+ return NOTIFY_OK;
+ }
+
+ ret = vfio_ap_mdev_open_once(matrix_mdev, data);
+ if (ret)
+ return NOTIFY_DONE;
+
+ matrix_mdev->kvm = data;
+
+ ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
+ if (ret)
+ return ret;
+
+ vfio_ap_mdev_copy_masks(matrix_mdev);
+
+ return NOTIFY_OK;
+}
+
+static int vfio_ap_mdev_open(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long events;
+ int ret;
+
+
+ if (!try_module_get(THIS_MODULE))
+ return -ENODEV;
+
+ matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
+ events = VFIO_GROUP_NOTIFY_SET_KVM;
+
+ ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &events, &matrix_mdev->group_notifier);
+ if (ret) {
+ module_put(THIS_MODULE);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void vfio_ap_mdev_release(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ if (matrix_mdev->kvm)
+ kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+
+ vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &matrix_mdev->group_notifier);
+ matrix_mdev->kvm = NULL;
+ module_put(THIS_MODULE);
+}
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
.mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
+ .open = vfio_ap_mdev_open,
+ .release = vfio_ap_mdev_release,
};
int vfio_ap_mdev_register(void)
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index a2eab78..ebad5db 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -71,10 +71,15 @@ struct ap_matrix {
* @list: allows the ap_matrix_mdev struct to be added to a list
* @matrix: the adapters, usage domains and control domains assigned to the
* mediated matrix device.
+ * @group_notifier: notifier block used for specifying callback function for
+ * handling the VFIO_GROUP_NOTIFY_SET_KVM event
+ * @kvm: the struct holding guest's state
*/
struct ap_matrix_mdev {
struct list_head node;
struct ap_matrix matrix;
+ struct notifier_block group_notifier;
+ struct kvm *kvm;
};
extern int vfio_ap_mdev_register(void);
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-12 19:43 ` [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback Tony Krowiak
@ 2018-09-18 17:00 ` Halil Pasic
2018-09-18 21:57 ` Tony Krowiak
2018-09-21 23:28 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: Halil Pasic @ 2018-09-18 17:00 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja, Tony Krowiak
On 09/12/2018 09:43 PM, Tony Krowiak wrote:
> +/**
> + * vfio_ap_mdev_open_once
> + *
> + * @matrix_mdev: a mediated matrix device
> + *
> + * Return 0 if no other mediated matrix device has been opened for the
> + * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
> + */
> +static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
> + struct kvm *kvm)
> +{
> + struct ap_matrix_mdev *m;
> +
> + mutex_lock(&matrix_dev->lock);
> +
> + list_for_each_entry(m, &matrix_dev->mdev_list, node) {
> + if ((m != matrix_mdev) && (m->kvm == kvm)) {
> + mutex_unlock(&matrix_dev->lock);
> + return -EPERM;
> + }
> + }
> +
> + mutex_unlock(&matrix_dev->lock);
> +
> + return 0;
> +}
> +
> +static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
> + unsigned long action, void *data)
> +{
> + int ret;
> + struct ap_matrix_mdev *matrix_mdev;
> +
> + if (action != VFIO_GROUP_NOTIFY_SET_KVM)
> + return NOTIFY_OK;
> +
> + matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
> +
> + if (!data) {
> + matrix_mdev->kvm = NULL;
> + return NOTIFY_OK;
> + }
> +
> + ret = vfio_ap_mdev_open_once(matrix_mdev, data);
This could be racy. Two threads doing vfio_ap_mdev_group_notifier()
can first get 0 here in a sense that there is no such kvm in the list,
and then both set the very same kvm three lines below. Which would
result in what we are trying to prevent.
Also vfio_ap_mdev_open_once() does not seem like an appropriate name
any more. If we were to do the matrix_mdev->kvm = kvm in there we could
call it something like vfio_ap_mdev_set_kvm().
> + if (ret)
> + return NOTIFY_DONE;
> +
> + matrix_mdev->kvm = data;
> +
> + ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
> + if (ret)
> + return ret;
> +
> + vfio_ap_mdev_copy_masks(matrix_mdev);
> +
> + return NOTIFY_OK;
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-18 17:00 ` Halil Pasic
@ 2018-09-18 21:57 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-18 21:57 UTC (permalink / raw)
To: Halil Pasic, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/18/2018 01:00 PM, Halil Pasic wrote:
>
> On 09/12/2018 09:43 PM, Tony Krowiak wrote:
>> +/**
>> + * vfio_ap_mdev_open_once
>> + *
>> + * @matrix_mdev: a mediated matrix device
>> + *
>> + * Return 0 if no other mediated matrix device has been opened for the
>> + * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
>> + */
>> +static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
>> + struct kvm *kvm)
>> +{
>> + struct ap_matrix_mdev *m;
>> +
>> + mutex_lock(&matrix_dev->lock);
>> +
>> + list_for_each_entry(m, &matrix_dev->mdev_list, node) {
>> + if ((m != matrix_mdev) && (m->kvm == kvm)) {
>> + mutex_unlock(&matrix_dev->lock);
>> + return -EPERM;
>> + }
>> + }
>> +
>> + mutex_unlock(&matrix_dev->lock);
>> +
>> + return 0;
>> +}
>> +
>> +static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
>> + unsigned long action, void *data)
>> +{
>> + int ret;
>> + struct ap_matrix_mdev *matrix_mdev;
>> +
>> + if (action != VFIO_GROUP_NOTIFY_SET_KVM)
>> + return NOTIFY_OK;
>> +
>> + matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
>> +
>> + if (!data) {
>> + matrix_mdev->kvm = NULL;
>> + return NOTIFY_OK;
>> + }
>> +
>> + ret = vfio_ap_mdev_open_once(matrix_mdev, data);
> This could be racy. Two threads doing vfio_ap_mdev_group_notifier()
> can first get 0 here in a sense that there is no such kvm in the list,
> and then both set the very same kvm three lines below. Which would
> result in what we are trying to prevent.
>
> Also vfio_ap_mdev_open_once() does not seem like an appropriate name
> any more. If we were to do the matrix_mdev->kvm = kvm in there we could
> call it something like vfio_ap_mdev_set_kvm().
I'm moving the matrix-mdev->kvm = kvm inside the mutex lock in
vfio_ap_mdev_open_once() ... also renaming it to vfio_ap_mdev_set_kvm().
>
>> + if (ret)
>> + return NOTIFY_DONE;
>> +
>> + matrix_mdev->kvm = data;
>> +
>> + ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
>> + if (ret)
>> + return ret;
>> +
>> + vfio_ap_mdev_copy_masks(matrix_mdev);
>> +
>> + return NOTIFY_OK;
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-12 19:43 ` [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback Tony Krowiak
2018-09-18 17:00 ` Halil Pasic
@ 2018-09-21 23:28 ` Tony Krowiak
2018-09-24 8:40 ` David Hildenbrand
1 sibling, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-21 23:28 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/12/2018 03:43 PM, Tony Krowiak wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Implements the open callback on the mediated matrix device.
> The function registers a group notifier to receive notification
> of the VFIO_GROUP_NOTIFY_SET_KVM event. When notified,
> the vfio_ap device driver will get access to the guest's
> kvm structure. The open callback must ensure that only one
> mediated device shall be opened per guest.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Acked-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Acked-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/include/asm/kvm_host.h | 1 +
> drivers/s390/crypto/vfio_ap_ops.c | 168 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 5 +
> 3 files changed, 174 insertions(+), 0 deletions(-)
>
(...)
> @@ -699,12 +730,149 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
> NULL
> };
>
> +/**
> + * Verify that the AP instructions are available on the guest. This is indicated
> + * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
> + */
> +static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
> +{
> + if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
> + return 0;
> +
> + return -EOPNOTSUPP;
> +}
> +
(...)
> +
> +/**
> + * vfio_ap_mdev_open_once
> + *
> + * @matrix_mdev: a mediated matrix device
> + *
> + * Return 0 if no other mediated matrix device has been opened for the
> + * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
> + */
> +static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
> + struct kvm *kvm)
> +{
> + struct ap_matrix_mdev *m;
> +
> + mutex_lock(&matrix_dev->lock);
> +
> + list_for_each_entry(m, &matrix_dev->mdev_list, node) {
> + if ((m != matrix_mdev) && (m->kvm == kvm)) {
> + mutex_unlock(&matrix_dev->lock);
> + return -EPERM;
> + }
> + }
> +
> + mutex_unlock(&matrix_dev->lock);
> +
> + return 0;
> +}
> +
> +static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
> + unsigned long action, void *data)
> +{
> + int ret;
> + struct ap_matrix_mdev *matrix_mdev;
> +
> + if (action != VFIO_GROUP_NOTIFY_SET_KVM)
> + return NOTIFY_OK;
> +
> + matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
> +
> + if (!data) {
> + matrix_mdev->kvm = NULL;
> + return NOTIFY_OK;
> + }
> +
> + ret = vfio_ap_mdev_open_once(matrix_mdev, data);
> + if (ret)
> + return NOTIFY_DONE;
> +
> + matrix_mdev->kvm = data;
> +
> + ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
> + if (ret)
> + return ret;
> +
> + vfio_ap_mdev_copy_masks(matrix_mdev);
> +
> + return NOTIFY_OK;
> +}
> +
> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
> +{
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> + unsigned long events;
> + int ret;
> +
> +
> + if (!try_module_get(THIS_MODULE))
> + return -ENODEV;
> +
> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
> + events = VFIO_GROUP_NOTIFY_SET_KVM;
> +
> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
> + &events, &matrix_mdev->group_notifier);
> + if (ret) {
> + module_put(THIS_MODULE);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
(...)
The fixup! patch below modifies this patch (11/26) to illustrate how
David's recommendation will be implemented for v11 of the series. It
is one of three fixup! patches (the other two are in responses to
03/26 and 25/26) included to generate discussion in v10 rather than
waiting until v11 for comments.
-----------------------------------8<-----------------------------------
From: Tony Krowiak <akrowiak@linux.ibm.com>
Date: Thu, 20 Sep 2018 12:01:53 -0400
Subject: [FIXUP v10] fixup!: s390: vfio-ap: implement mediated device
open callback
* Fix race condition in KVM notifier
* Remove test for KVM_S390_VM_CPU_FEAT_AP
---
drivers/s390/crypto/vfio_ap_ops.c | 26 +++++++++++++++-----------
1 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c
b/drivers/s390/crypto/vfio_ap_ops.c
index 8bc0cdd..573a5cc 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -731,12 +731,12 @@ static ssize_t matrix_show(struct device *dev,
struct device_attribute *attr,
};
/**
- * Verify that the AP instructions are available on the guest. This is
indicated
- * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
+ * Verify that the AP instructions are being interpreted by firmware
for the
+ * guest. This is indicated by the kvm->arch.crypto.apie flag.
*/
static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
{
- if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
+ if (kvm->arch.crypto.apie)
return 0;
return -EOPNOTSUPP;
@@ -772,15 +772,19 @@ static void vfio_ap_mdev_copy_masks(struct
ap_matrix_mdev *matrix_mdev)
}
/**
- * vfio_ap_mdev_open_once
+ * vfio_ap_mdev_set_kvm
*
* @matrix_mdev: a mediated matrix device
+ * @kvm: reference to KVM instance
*
- * Return 0 if no other mediated matrix device has been opened for the
- * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
+ * Verifies no other mediated matrix device has a reference to @kvm and
sets a
+ * reference to it in @matrix_mdev->kvm.
+ *
+ * Return 0 if no other mediated matrix device has a reference to @kvm;
+ * otherwise, returns -EPERM.
*/
-static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
- struct kvm *kvm)
+static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
+ struct kvm *kvm)
{
struct ap_matrix_mdev *m;
@@ -793,6 +797,7 @@ static int vfio_ap_mdev_open_once(struct
ap_matrix_mdev *matrix_mdev,
}
}
+ matrix_mdev->kvm = kvm;
mutex_unlock(&matrix_dev->lock);
return 0;
@@ -814,16 +819,15 @@ static int vfio_ap_mdev_group_notifier(struct
notifier_block *nb,
return NOTIFY_OK;
}
- ret = vfio_ap_mdev_open_once(matrix_mdev, data);
+ ret = vfio_ap_mdev_set_kvm(matrix_mdev, data);
if (ret)
return NOTIFY_DONE;
- matrix_mdev->kvm = data;
-
ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
if (ret)
return ret;
+
vfio_ap_mdev_copy_masks(matrix_mdev);
return NOTIFY_OK;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-21 23:28 ` Tony Krowiak
@ 2018-09-24 8:40 ` David Hildenbrand
2018-09-24 16:07 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 8:40 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
> /**
> - * Verify that the AP instructions are available on the guest. This is
> indicated
> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
> + * Verify that the AP instructions are being interpreted by firmware
> for the
> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
> */
> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
> {
> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
> + if (kvm->arch.crypto.apie)
> return 0;
I wonder if this check makes sense, because apie can be toggled during
runtime. I guess it would be sufficient to check if the ap control block
is available and apie is supported by the HW.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 8:40 ` David Hildenbrand
@ 2018-09-24 16:07 ` Tony Krowiak
2018-09-24 18:40 ` David Hildenbrand
0 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:07 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>
>> /**
>> - * Verify that the AP instructions are available on the guest. This is
>> indicated
>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>> + * Verify that the AP instructions are being interpreted by firmware
>> for the
>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>> */
>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>> {
>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>> + if (kvm->arch.crypto.apie)
>> return 0;
>
> I wonder if this check makes sense, because apie can be toggled during
> runtime. I guess it would be sufficient to check if the ap control block
> is available and apie is supported by the HW.
I am not clear about what you are getting at here, but I'll attempt
to respond. There is no need to check if the AP control block (CRYCB)
is available as the address is set in the CRYCBD three instructions
above, even if AP instructions are not available. Regarding whether apie
is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
can not be set unless it is supported by the HW. In the patch (24/26)
that provides the VM attributes to toggle this value, it can only be
turned on if the AP instructions are available. I might also note that
the kvm_ap_validate_crypto_setup() function is called whenever one of
the VM crypto attributes is changed, so it makes sense that decisions
made in this function are based on a change to a VM crypto attribute. In
my first pass at changing this function, I checked
ap_instructions_available() here, but after considering all of the
above, it made sense to me to check the apie flag.
>
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 16:07 ` Tony Krowiak
@ 2018-09-24 18:40 ` David Hildenbrand
2018-09-24 18:43 ` Tony Krowiak
2018-09-24 19:46 ` Tony Krowiak
0 siblings, 2 replies; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 18:40 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 24/09/2018 18:07, Tony Krowiak wrote:
> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>
>>> /**
>>> - * Verify that the AP instructions are available on the guest. This is
>>> indicated
>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>> + * Verify that the AP instructions are being interpreted by firmware
>>> for the
>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>> */
>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>> {
>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>> + if (kvm->arch.crypto.apie)
>>> return 0;
>>
>> I wonder if this check makes sense, because apie can be toggled during
>> runtime. I guess it would be sufficient to check if the ap control block
>> is available and apie is supported by the HW.
>
> I am not clear about what you are getting at here, but I'll attempt
> to respond. There is no need to check if the AP control block (CRYCB)
> is available as the address is set in the CRYCBD three instructions
> above, even if AP instructions are not available. Regarding whether apie
> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
> can not be set unless it is supported by the HW. In the patch (24/26)
> that provides the VM attributes to toggle this value, it can only be
> turned on if the AP instructions are available. I might also note that
> the kvm_ap_validate_crypto_setup() function is called whenever one of
> the VM crypto attributes is changed, so it makes sense that decisions
> made in this function are based on a change to a VM crypto attribute. In
> my first pass at changing this function, I checked
> ap_instructions_available() here, but after considering all of the
> above, it made sense to me to check the apie flag.
>
I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
is a moving target.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 18:40 ` David Hildenbrand
@ 2018-09-24 18:43 ` Tony Krowiak
2018-09-24 19:46 ` Tony Krowiak
1 sibling, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 18:43 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 02:40 PM, David Hildenbrand wrote:
> On 24/09/2018 18:07, Tony Krowiak wrote:
>> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>>
>>>> /**
>>>> - * Verify that the AP instructions are available on the guest. This is
>>>> indicated
>>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>>> + * Verify that the AP instructions are being interpreted by firmware
>>>> for the
>>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>>> */
>>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>>> {
>>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>>> + if (kvm->arch.crypto.apie)
>>>> return 0;
>>>
>>> I wonder if this check makes sense, because apie can be toggled during
>>> runtime. I guess it would be sufficient to check if the ap control block
>>> is available and apie is supported by the HW.
>>
>> I am not clear about what you are getting at here, but I'll attempt
>> to respond. There is no need to check if the AP control block (CRYCB)
>> is available as the address is set in the CRYCBD three instructions
>> above, even if AP instructions are not available. Regarding whether apie
>> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
>> can not be set unless it is supported by the HW. In the patch (24/26)
>> that provides the VM attributes to toggle this value, it can only be
>> turned on if the AP instructions are available. I might also note that
>> the kvm_ap_validate_crypto_setup() function is called whenever one of
>> the VM crypto attributes is changed, so it makes sense that decisions
>> made in this function are based on a change to a VM crypto attribute. In
>> my first pass at changing this function, I checked
>> ap_instructions_available() here, but after considering all of the
>> above, it made sense to me to check the apie flag.
>>
>
> I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
> is a moving target.
Okay then.
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 18:40 ` David Hildenbrand
2018-09-24 18:43 ` Tony Krowiak
@ 2018-09-24 19:46 ` Tony Krowiak
2018-09-24 19:55 ` David Hildenbrand
1 sibling, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 19:46 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 02:40 PM, David Hildenbrand wrote:
> On 24/09/2018 18:07, Tony Krowiak wrote:
>> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>>
>>>> /**
>>>> - * Verify that the AP instructions are available on the guest. This is
>>>> indicated
>>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>>> + * Verify that the AP instructions are being interpreted by firmware
>>>> for the
>>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>>> */
>>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>>> {
>>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>>> + if (kvm->arch.crypto.apie)
>>>> return 0;
>>>
>>> I wonder if this check makes sense, because apie can be toggled during
>>> runtime. I guess it would be sufficient to check if the ap control block
>>> is available and apie is supported by the HW.
>>
>> I am not clear about what you are getting at here, but I'll attempt
>> to respond. There is no need to check if the AP control block (CRYCB)
>> is available as the address is set in the CRYCBD three instructions
>> above, even if AP instructions are not available. Regarding whether apie
>> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
>> can not be set unless it is supported by the HW. In the patch (24/26)
>> that provides the VM attributes to toggle this value, it can only be
>> turned on if the AP instructions are available. I might also note that
>> the kvm_ap_validate_crypto_setup() function is called whenever one of
>> the VM crypto attributes is changed, so it makes sense that decisions
>> made in this function are based on a change to a VM crypto attribute. In
>> my first pass at changing this function, I checked
>> ap_instructions_available() here, but after considering all of the
>> above, it made sense to me to check the apie flag.
>>
>
> I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
> is a moving target.
Looking at this again, I think I responded before my brain shifted from
digesting comments about patch 24/26 (enable/disable APIE) to the
context for your comment here; namely, the device open callback. My
comment above makes no sense in this context. From the perspective of
the vfio_ap device driver, there is one requirement that must be met in
order to provide pass-through functionality: The AP instructions must be
must be interpreted by the HW (i.e., ECA.28 == 1). Checking whether AP
instructions are available does not tell us whether they are being
interpreted by HW. Checking whether the AP control block (i.e., CRYCB)
is available, even when combined with the instruction availability
check, does not provide any more insight into the value of ECA.28
becuase the CRYCB will be provided if the MSAX3 facility is installed
(STFLE.76) for the guest regardless of whether AP instructions are
available or not. There is no doubt that if the AP instructions are
not available, then the mdev open callback should fail, but it doesn't
tell the whole story.
I realize that our CPU model protects against configuring a vfio-ap
device for the guest if ap=off, but this function knows nothing about
userspace. I can make a similar argument that kvm->arch.crypto.apie
will be switched on only if ap=on but again, that is userspace
configuration.
Having said all of the above, maybe it doesn't really matter whether
AP instructions are being interpreted or not. If ECA.28 == 0, then
the AP masks may very well be ignored since all AP instructions will
be intercepted; so, maybe checking AP instruction availability is all
that is needed. I will verify this and if I'm correct, I'll make the
change you suggested.
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 19:46 ` Tony Krowiak
@ 2018-09-24 19:55 ` David Hildenbrand
2018-09-25 19:54 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 19:55 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 24/09/2018 21:46, Tony Krowiak wrote:
> On 09/24/2018 02:40 PM, David Hildenbrand wrote:
>> On 24/09/2018 18:07, Tony Krowiak wrote:
>>> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>>>
>>>>> /**
>>>>> - * Verify that the AP instructions are available on the guest. This is
>>>>> indicated
>>>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>>>> + * Verify that the AP instructions are being interpreted by firmware
>>>>> for the
>>>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>>>> */
>>>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>>>> {
>>>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>>>> + if (kvm->arch.crypto.apie)
>>>>> return 0;
>>>>
>>>> I wonder if this check makes sense, because apie can be toggled during
>>>> runtime. I guess it would be sufficient to check if the ap control block
>>>> is available and apie is supported by the HW.
>>>
>>> I am not clear about what you are getting at here, but I'll attempt
>>> to respond. There is no need to check if the AP control block (CRYCB)
>>> is available as the address is set in the CRYCBD three instructions
>>> above, even if AP instructions are not available. Regarding whether apie
>>> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
>>> can not be set unless it is supported by the HW. In the patch (24/26)
>>> that provides the VM attributes to toggle this value, it can only be
>>> turned on if the AP instructions are available. I might also note that
>>> the kvm_ap_validate_crypto_setup() function is called whenever one of
>>> the VM crypto attributes is changed, so it makes sense that decisions
>>> made in this function are based on a change to a VM crypto attribute. In
>>> my first pass at changing this function, I checked
>>> ap_instructions_available() here, but after considering all of the
>>> above, it made sense to me to check the apie flag.
>>>
>>
>> I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
>> is a moving target.
>
> Looking at this again, I think I responded before my brain shifted from
> digesting comments about patch 24/26 (enable/disable APIE) to the
> context for your comment here; namely, the device open callback. My
> comment above makes no sense in this context. From the perspective of
> the vfio_ap device driver, there is one requirement that must be met in
> order to provide pass-through functionality: The AP instructions must be
> must be interpreted by the HW (i.e., ECA.28 == 1). Checking whether AP
> instructions are available does not tell us whether they are being
> interpreted by HW. Checking whether the AP control block (i.e., CRYCB)
> is available, even when combined with the instruction availability
> check, does not provide any more insight into the value of ECA.28
> becuase the CRYCB will be provided if the MSAX3 facility is installed
> (STFLE.76) for the guest regardless of whether AP instructions are
> available or not. There is no doubt that if the AP instructions are
> not available, then the mdev open callback should fail, but it doesn't
> tell the whole story.
>
> I realize that our CPU model protects against configuring a vfio-ap
> device for the guest if ap=off, but this function knows nothing about
> userspace. I can make a similar argument that kvm->arch.crypto.apie
> will be switched on only if ap=on but again, that is userspace
> configuration.
>
> Having said all of the above, maybe it doesn't really matter whether
> AP instructions are being interpreted or not. If ECA.28 == 0, then
> the AP masks may very well be ignored since all AP instructions will
> be intercepted; so, maybe checking AP instruction availability is all
> that is needed. I will verify this and if I'm correct, I'll make the
> change you suggested.
Yes, that was exactly what I had in mind - we just have to make sure
that the ap control block exists, so we can set the right mask bits. If
QEMU asks for an intercept, it shall get an intercept.
But please proceed with whatever you think is best!
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-24 19:55 ` David Hildenbrand
@ 2018-09-25 19:54 ` Tony Krowiak
2018-09-25 19:55 ` David Hildenbrand
0 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-25 19:54 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 03:55 PM, David Hildenbrand wrote:
> On 24/09/2018 21:46, Tony Krowiak wrote:
>> On 09/24/2018 02:40 PM, David Hildenbrand wrote:
>>> On 24/09/2018 18:07, Tony Krowiak wrote:
>>>> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>>>>
>>>>>> /**
>>>>>> - * Verify that the AP instructions are available on the guest. This is
>>>>>> indicated
>>>>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>>>>> + * Verify that the AP instructions are being interpreted by firmware
>>>>>> for the
>>>>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>>>>> */
>>>>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>>>>> {
>>>>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>>>>> + if (kvm->arch.crypto.apie)
>>>>>> return 0;
>>>>>
>>>>> I wonder if this check makes sense, because apie can be toggled during
>>>>> runtime. I guess it would be sufficient to check if the ap control block
>>>>> is available and apie is supported by the HW.
>>>>
>>>> I am not clear about what you are getting at here, but I'll attempt
>>>> to respond. There is no need to check if the AP control block (CRYCB)
>>>> is available as the address is set in the CRYCBD three instructions
>>>> above, even if AP instructions are not available. Regarding whether apie
>>>> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
>>>> can not be set unless it is supported by the HW. In the patch (24/26)
>>>> that provides the VM attributes to toggle this value, it can only be
>>>> turned on if the AP instructions are available. I might also note that
>>>> the kvm_ap_validate_crypto_setup() function is called whenever one of
>>>> the VM crypto attributes is changed, so it makes sense that decisions
>>>> made in this function are based on a change to a VM crypto attribute. In
>>>> my first pass at changing this function, I checked
>>>> ap_instructions_available() here, but after considering all of the
>>>> above, it made sense to me to check the apie flag.
>>>>
>>>
>>> I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
>>> is a moving target.
>>
>> Looking at this again, I think I responded before my brain shifted from
>> digesting comments about patch 24/26 (enable/disable APIE) to the
>> context for your comment here; namely, the device open callback. My
>> comment above makes no sense in this context. From the perspective of
>> the vfio_ap device driver, there is one requirement that must be met in
>> order to provide pass-through functionality: The AP instructions must be
>> must be interpreted by the HW (i.e., ECA.28 == 1). Checking whether AP
>> instructions are available does not tell us whether they are being
>> interpreted by HW. Checking whether the AP control block (i.e., CRYCB)
>> is available, even when combined with the instruction availability
>> check, does not provide any more insight into the value of ECA.28
>> becuase the CRYCB will be provided if the MSAX3 facility is installed
>> (STFLE.76) for the guest regardless of whether AP instructions are
>> available or not. There is no doubt that if the AP instructions are
>> not available, then the mdev open callback should fail, but it doesn't
>> tell the whole story.
>>
>> I realize that our CPU model protects against configuring a vfio-ap
>> device for the guest if ap=off, but this function knows nothing about
>> userspace. I can make a similar argument that kvm->arch.crypto.apie
>> will be switched on only if ap=on but again, that is userspace
>> configuration.
>>
>> Having said all of the above, maybe it doesn't really matter whether
>> AP instructions are being interpreted or not. If ECA.28 == 0, then
>> the AP masks may very well be ignored since all AP instructions will
>> be intercepted; so, maybe checking AP instruction availability is all
>> that is needed. I will verify this and if I'm correct, I'll make the
>> change you suggested.
>
> Yes, that was exactly what I had in mind - we just have to make sure
> that the ap control block exists, so we can set the right mask bits. If
> QEMU asks for an intercept, it shall get an intercept.
>
> But please proceed with whatever you think is best!
After discussing this with Halil, here's what I decided:
* There will be no check for kvm->arch.crypto.apie here
* A check for ap_instructions_available() will not be executed
here, but inserted into the vfio_ap module init function.
The module init function will fail (ENODEV) if the AP
instructions are not installed. In my (our) opinion that
makes more sense given the purpose of the vfio_ap driver is
to pass through the AP instructions to the guest.
* A check will be added here to verify the CRYCB is available (i.e.,
matrix_mdev->kvm->arch.crypto.crycbd != 0).
>
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback
2018-09-25 19:54 ` Tony Krowiak
@ 2018-09-25 19:55 ` David Hildenbrand
0 siblings, 0 replies; 87+ messages in thread
From: David Hildenbrand @ 2018-09-25 19:55 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 25/09/2018 21:54, Tony Krowiak wrote:
> On 09/24/2018 03:55 PM, David Hildenbrand wrote:
>> On 24/09/2018 21:46, Tony Krowiak wrote:
>>> On 09/24/2018 02:40 PM, David Hildenbrand wrote:
>>>> On 24/09/2018 18:07, Tony Krowiak wrote:
>>>>> On 09/24/2018 04:40 AM, David Hildenbrand wrote:
>>>>>>
>>>>>>> /**
>>>>>>> - * Verify that the AP instructions are available on the guest. This is
>>>>>>> indicated
>>>>>>> - * via the KVM_S390_VM_CPU_FEAT_AP CPU model feature.
>>>>>>> + * Verify that the AP instructions are being interpreted by firmware
>>>>>>> for the
>>>>>>> + * guest. This is indicated by the kvm->arch.crypto.apie flag.
>>>>>>> */
>>>>>>> static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
>>>>>>> {
>>>>>>> - if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat))
>>>>>>> + if (kvm->arch.crypto.apie)
>>>>>>> return 0;
>>>>>>
>>>>>> I wonder if this check makes sense, because apie can be toggled during
>>>>>> runtime. I guess it would be sufficient to check if the ap control block
>>>>>> is available and apie is supported by the HW.
>>>>>
>>>>> I am not clear about what you are getting at here, but I'll attempt
>>>>> to respond. There is no need to check if the AP control block (CRYCB)
>>>>> is available as the address is set in the CRYCBD three instructions
>>>>> above, even if AP instructions are not available. Regarding whether apie
>>>>> is supported by the hardware, the value of vcpu->kvm->arch.crypto.apie
>>>>> can not be set unless it is supported by the HW. In the patch (24/26)
>>>>> that provides the VM attributes to toggle this value, it can only be
>>>>> turned on if the AP instructions are available. I might also note that
>>>>> the kvm_ap_validate_crypto_setup() function is called whenever one of
>>>>> the VM crypto attributes is changed, so it makes sense that decisions
>>>>> made in this function are based on a change to a VM crypto attribute. In
>>>>> my first pass at changing this function, I checked
>>>>> ap_instructions_available() here, but after considering all of the
>>>>> above, it made sense to me to check the apie flag.
>>>>>
>>>>
>>>> I prefer ap_instructions_available(). As I said, kvm->arch.crypto.apie
>>>> is a moving target.
>>>
>>> Looking at this again, I think I responded before my brain shifted from
>>> digesting comments about patch 24/26 (enable/disable APIE) to the
>>> context for your comment here; namely, the device open callback. My
>>> comment above makes no sense in this context. From the perspective of
>>> the vfio_ap device driver, there is one requirement that must be met in
>>> order to provide pass-through functionality: The AP instructions must be
>>> must be interpreted by the HW (i.e., ECA.28 == 1). Checking whether AP
>>> instructions are available does not tell us whether they are being
>>> interpreted by HW. Checking whether the AP control block (i.e., CRYCB)
>>> is available, even when combined with the instruction availability
>>> check, does not provide any more insight into the value of ECA.28
>>> becuase the CRYCB will be provided if the MSAX3 facility is installed
>>> (STFLE.76) for the guest regardless of whether AP instructions are
>>> available or not. There is no doubt that if the AP instructions are
>>> not available, then the mdev open callback should fail, but it doesn't
>>> tell the whole story.
>>>
>>> I realize that our CPU model protects against configuring a vfio-ap
>>> device for the guest if ap=off, but this function knows nothing about
>>> userspace. I can make a similar argument that kvm->arch.crypto.apie
>>> will be switched on only if ap=on but again, that is userspace
>>> configuration.
>>>
>>> Having said all of the above, maybe it doesn't really matter whether
>>> AP instructions are being interpreted or not. If ECA.28 == 0, then
>>> the AP masks may very well be ignored since all AP instructions will
>>> be intercepted; so, maybe checking AP instruction availability is all
>>> that is needed. I will verify this and if I'm correct, I'll make the
>>> change you suggested.
>>
>> Yes, that was exactly what I had in mind - we just have to make sure
>> that the ap control block exists, so we can set the right mask bits. If
>> QEMU asks for an intercept, it shall get an intercept.
>>
>> But please proceed with whatever you think is best!
>
> After discussing this with Halil, here's what I decided:
> * There will be no check for kvm->arch.crypto.apie here
> * A check for ap_instructions_available() will not be executed
> here, but inserted into the vfio_ap module init function.
> The module init function will fail (ENODEV) if the AP
> instructions are not installed. In my (our) opinion that
> makes more sense given the purpose of the vfio_ap driver is
> to pass through the AP instructions to the guest.
> * A check will be added here to verify the CRYCB is available (i.e.,
> matrix_mdev->kvm->arch.crypto.crycbd != 0).
>
Sounds good to me!
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (10 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 11/26] s390: vfio-ap: implement mediated device open callback Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-24 11:43 ` Cornelia Huck
2018-09-12 19:43 ` [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues Tony Krowiak
` (15 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces ioctl access to the VFIO AP Matrix device driver
by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
provides the VFIO AP Matrix device driver information to the
guest machine.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Acked-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 38 +++++++++++++++++++++++++++++++++++++
include/uapi/linux/vfio.h | 1 +
2 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 8bc0cdd..f8b276a 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -865,6 +865,43 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
module_put(THIS_MODULE);
}
+static int vfio_ap_mdev_get_device_info(unsigned long arg)
+{
+ unsigned long minsz;
+ struct vfio_device_info info;
+
+ minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz)
+ return -EINVAL;
+
+ info.flags = VFIO_DEVICE_FLAGS_AP;
+ info.num_regions = 0;
+ info.num_irqs = 0;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+}
+
+static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
+ unsigned int cmd, unsigned long arg)
+{
+ int ret;
+
+ switch (cmd) {
+ case VFIO_DEVICE_GET_INFO:
+ ret = vfio_ap_mdev_get_device_info(arg);
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
@@ -873,6 +910,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
.remove = vfio_ap_mdev_remove,
.open = vfio_ap_mdev_open,
.release = vfio_ap_mdev_release,
+ .ioctl = vfio_ap_mdev_ioctl,
};
int vfio_ap_mdev_register(void)
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index bfbe2be..f378b98 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -200,6 +200,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
#define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
#define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
};
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
2018-09-12 19:43 ` [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
@ 2018-09-24 11:43 ` Cornelia Huck
2018-09-24 16:29 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:43 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:43:02 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces ioctl access to the VFIO AP Matrix device driver
> by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
> provides the VFIO AP Matrix device driver information to the
> guest machine.
What about the following?
"Add support for the VFIO_DEVICE_GET_INFO ioctl to the VFIO AP Matrix
device driver. This is a minimal implementation, as vfio-ap does not
use I/O regions."
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
> Acked-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 38 +++++++++++++++++++++++++++++++++++++
> include/uapi/linux/vfio.h | 1 +
> 2 files changed, 39 insertions(+), 0 deletions(-)
Anyway,
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
2018-09-24 11:43 ` Cornelia Huck
@ 2018-09-24 16:29 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:29 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/24/2018 07:43 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:43:02 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces ioctl access to the VFIO AP Matrix device driver
>> by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
>> provides the VFIO AP Matrix device driver information to the
>> guest machine.
>
> What about the following?
>
> "Add support for the VFIO_DEVICE_GET_INFO ioctl to the VFIO AP Matrix
> device driver. This is a minimal implementation, as vfio-ap does not
> use I/O regions."
Sounds good, I'll change it.
>
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
>> Acked-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> drivers/s390/crypto/vfio_ap_ops.c | 38 +++++++++++++++++++++++++++++++++++++
>> include/uapi/linux/vfio.h | 1 +
>> 2 files changed, 39 insertions(+), 0 deletions(-)
>
> Anyway,
>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (11 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 12/26] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-24 11:36 ` Cornelia Huck
2018-09-12 19:43 ` [PATCH v10 14/26] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl Tony Krowiak
` (14 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Let's call PAPQ(ZAPQ) to zeroize a queue for each queue configured
for a mediated matrix device when it is released.
Zeroizing a queue resets the queue, clears all pending
messages for the queue entries and disables adapter interruptions
associated with the queue.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 44 +++++++++++++++++++++++++++++++++++++
1 files changed, 44 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index f8b276a..48b1b78 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -829,6 +829,49 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
return NOTIFY_OK;
}
+static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
+ unsigned int retry)
+{
+ struct ap_queue_status status;
+
+ do {
+ status = ap_zapq(AP_MKQID(apid, apqi));
+ switch (status.response_code) {
+ case AP_RESPONSE_NORMAL:
+ return 0;
+ case AP_RESPONSE_RESET_IN_PROGRESS:
+ case AP_RESPONSE_BUSY:
+ msleep(20);
+ break;
+ default:
+ /* things are really broken, give up */
+ return -EIO;
+ }
+ } while (retry--);
+
+ return -EBUSY;
+}
+
+static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
+{
+ int ret;
+ int rc = 0;
+ unsigned long apid, apqi;
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
+ matrix_mdev->matrix.apm_max + 1) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+ matrix_mdev->matrix.aqm_max + 1) {
+ ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
+ if (ret)
+ rc = ret;
+ }
+ }
+
+ return rc;
+}
+
static int vfio_ap_mdev_open(struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
@@ -859,6 +902,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
if (matrix_mdev->kvm)
kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+ vfio_ap_mdev_reset_queues(mdev);
vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
&matrix_mdev->group_notifier);
matrix_mdev->kvm = NULL;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-12 19:43 ` [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues Tony Krowiak
@ 2018-09-24 11:36 ` Cornelia Huck
2018-09-24 12:16 ` Halil Pasic
0 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:36 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:43:03 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Let's call PAPQ(ZAPQ) to zeroize a queue for each queue configured
> for a mediated matrix device when it is released.
>
> Zeroizing a queue resets the queue, clears all pending
> messages for the queue entries and disables adapter interruptions
> associated with the queue.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 44 +++++++++++++++++++++++++++++++++++++
> 1 files changed, 44 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index f8b276a..48b1b78 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -829,6 +829,49 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
> return NOTIFY_OK;
> }
>
> +static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
> + unsigned int retry)
> +{
> + struct ap_queue_status status;
> +
> + do {
> + status = ap_zapq(AP_MKQID(apid, apqi));
> + switch (status.response_code) {
> + case AP_RESPONSE_NORMAL:
> + return 0;
> + case AP_RESPONSE_RESET_IN_PROGRESS:
> + case AP_RESPONSE_BUSY:
> + msleep(20);
> + break;
> + default:
> + /* things are really broken, give up */
> + return -EIO;
> + }
> + } while (retry--);
> +
> + return -EBUSY;
So, this function may either return 0, -EIO (things are really broken),
or -EBUSY (still busy after multiple tries)...
> +}
> +
> +static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
> +{
> + int ret;
> + int rc = 0;
> + unsigned long apid, apqi;
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> +
> + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
> + matrix_mdev->matrix.apm_max + 1) {
> + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
> + matrix_mdev->matrix.aqm_max + 1) {
> + ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
> + if (ret)
> + rc = ret;
...and here, we return the last error of any of the resets. Two
questions:
- Does it make sense to continue if we get -EIO? IOW, does "really
broken" only refer to a certain tuple and other tuples still can/need
to be reset?
- Is the return code useful in any way, as we don't know which tuple it
refers to?
> + }
> + }
> +
> + return rc;
> +}
> +
> static int vfio_ap_mdev_open(struct mdev_device *mdev)
> {
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> @@ -859,6 +902,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
> if (matrix_mdev->kvm)
> kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>
> + vfio_ap_mdev_reset_queues(mdev);
> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
> &matrix_mdev->group_notifier);
> matrix_mdev->kvm = NULL;
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-24 11:36 ` Cornelia Huck
@ 2018-09-24 12:16 ` Halil Pasic
2018-09-24 12:32 ` Cornelia Huck
2018-09-24 13:22 ` Harald Freudenberger
0 siblings, 2 replies; 87+ messages in thread
From: Halil Pasic @ 2018-09-24 12:16 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On 09/24/2018 01:36 PM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:43:03 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Let's call PAPQ(ZAPQ) to zeroize a queue for each queue configured
>> for a mediated matrix device when it is released.
>>
>> Zeroizing a queue resets the queue, clears all pending
>> messages for the queue entries and disables adapter interruptions
>> associated with the queue.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> drivers/s390/crypto/vfio_ap_ops.c | 44 +++++++++++++++++++++++++++++++++++++
>> 1 files changed, 44 insertions(+), 0 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
>> index f8b276a..48b1b78 100644
>> --- a/drivers/s390/crypto/vfio_ap_ops.c
>> +++ b/drivers/s390/crypto/vfio_ap_ops.c
>> @@ -829,6 +829,49 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
>> return NOTIFY_OK;
>> }
>>
>> +static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
>> + unsigned int retry)
>> +{
>> + struct ap_queue_status status;
>> +
>> + do {
>> + status = ap_zapq(AP_MKQID(apid, apqi));
>> + switch (status.response_code) {
>> + case AP_RESPONSE_NORMAL:
>> + return 0;
>> + case AP_RESPONSE_RESET_IN_PROGRESS:
>> + case AP_RESPONSE_BUSY:
>> + msleep(20);
>> + break;
>> + default:
>> + /* things are really broken, give up */
>> + return -EIO;
>> + }
>> + } while (retry--);
>> +
>> + return -EBUSY;
>
> So, this function may either return 0, -EIO (things are really broken),
> or -EBUSY (still busy after multiple tries)...
>
>> +}
>> +
>> +static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
>> +{
>> + int ret;
>> + int rc = 0;
>> + unsigned long apid, apqi;
>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>> +
>> + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
>> + matrix_mdev->matrix.apm_max + 1) {
>> + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
>> + matrix_mdev->matrix.aqm_max + 1) {
>> + ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
>> + if (ret)
>> + rc = ret;
>
> ...and here, we return the last error of any of the resets. Two
> questions:
>
> - Does it make sense to continue if we get -EIO? IOW, does "really
> broken" only refer to a certain tuple and other tuples still can/need
> to be reset?
I think it does make sense to continue, because IMHO "things are really
broken" is an overstatement (I mean the APQN invalid case). One could
argue would skipping the current card (adapter) be justified or not.
IMHO the current code is good enough for the first shot, and we can think
about fine-tuning it later.
> - Is the return code useful in any way, as we don't know which tuple it
> refers to?
>
Well, good question. It conveys that the operation can 'fail'. AFAIR -EBUSY
is mostly fine given what the architecture say if we are satisfied with just
reset. And the cases behind -EIO might actually be OK too in the same sense.
My guess is, that based on the return value client code can tell if we have
zeroize for all queues or basically just reset (like rapq). We could log that
to some debug facility or whatever -- I guess, but at the moment we don't care.
In the end I think the code is good enough as is, and if we want we can
improve on it later.
Regards,
Halil
>> + }
>> + }
>> +
>> + return rc;
>> +}
>> +
>> static int vfio_ap_mdev_open(struct mdev_device *mdev)
>> {
>> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>> @@ -859,6 +902,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
>> if (matrix_mdev->kvm)
>> kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>>
>> + vfio_ap_mdev_reset_queues(mdev);
>> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>> &matrix_mdev->group_notifier);
>> matrix_mdev->kvm = NULL;
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-24 12:16 ` Halil Pasic
@ 2018-09-24 12:32 ` Cornelia Huck
2018-09-24 13:22 ` Harald Freudenberger
1 sibling, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 12:32 UTC (permalink / raw)
To: Halil Pasic
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Mon, 24 Sep 2018 14:16:42 +0200
Halil Pasic <pasic@linux.ibm.com> wrote:
> On 09/24/2018 01:36 PM, Cornelia Huck wrote:
> > On Wed, 12 Sep 2018 15:43:03 -0400
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> >> index f8b276a..48b1b78 100644
> >> --- a/drivers/s390/crypto/vfio_ap_ops.c
> >> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> >> @@ -829,6 +829,49 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
> >> return NOTIFY_OK;
> >> }
> >>
> >> +static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
> >> + unsigned int retry)
> >> +{
> >> + struct ap_queue_status status;
> >> +
> >> + do {
> >> + status = ap_zapq(AP_MKQID(apid, apqi));
> >> + switch (status.response_code) {
> >> + case AP_RESPONSE_NORMAL:
> >> + return 0;
> >> + case AP_RESPONSE_RESET_IN_PROGRESS:
> >> + case AP_RESPONSE_BUSY:
> >> + msleep(20);
> >> + break;
> >> + default:
> >> + /* things are really broken, give up */
> >> + return -EIO;
> >> + }
> >> + } while (retry--);
> >> +
> >> + return -EBUSY;
> >
> > So, this function may either return 0, -EIO (things are really broken),
> > or -EBUSY (still busy after multiple tries)...
> >
> >> +}
> >> +
> >> +static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
> >> +{
> >> + int ret;
> >> + int rc = 0;
> >> + unsigned long apid, apqi;
> >> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> >> +
> >> + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
> >> + matrix_mdev->matrix.apm_max + 1) {
> >> + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
> >> + matrix_mdev->matrix.aqm_max + 1) {
> >> + ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
> >> + if (ret)
> >> + rc = ret;
> >
> > ...and here, we return the last error of any of the resets. Two
> > questions:
> >
> > - Does it make sense to continue if we get -EIO? IOW, does "really
> > broken" only refer to a certain tuple and other tuples still can/need
> > to be reset?
>
> I think it does make sense to continue, because IMHO "things are really
> broken" is an overstatement (I mean the APQN invalid case). One could
> argue would skipping the current card (adapter) be justified or not.
A short comment ("even after -EIO, other devices still need to be
reset") may be helpful here (remember that I don't have any way to
verify this with the architecture).
>
> IMHO the current code is good enough for the first shot, and we can think
> about fine-tuning it later.
Sure.
>
> > - Is the return code useful in any way, as we don't know which tuple it
> > refers to?
> >
>
> Well, good question. It conveys that the operation can 'fail'. AFAIR -EBUSY
> is mostly fine given what the architecture say if we are satisfied with just
> reset. And the cases behind -EIO might actually be OK too in the same sense.
> My guess is, that based on the return value client code can tell if we have
> zeroize for all queues or basically just reset (like rapq). We could log that
> to some debug facility or whatever -- I guess, but at the moment we don't care.
Logging would probably be more useful than the return code, but that
can be added later.
>
> In the end I think the code is good enough as is, and if we want we can
> improve on it later.
I don't object to that; but this is all a bit confusing to readers
without access to the architecture, so I think a comment or two would
really improve things.
>
> Regards,
> Halil
>
>
> >> + }
> >> + }
> >> +
> >> + return rc;
> >> +}
> >> +
> >> static int vfio_ap_mdev_open(struct mdev_device *mdev)
> >> {
> >> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> >> @@ -859,6 +902,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
> >> if (matrix_mdev->kvm)
> >> kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> >>
> >> + vfio_ap_mdev_reset_queues(mdev);
> >> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
> >> &matrix_mdev->group_notifier);
> >> matrix_mdev->kvm = NULL;
> >
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-24 12:16 ` Halil Pasic
2018-09-24 12:32 ` Cornelia Huck
@ 2018-09-24 13:22 ` Harald Freudenberger
2018-09-24 16:42 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: Harald Freudenberger @ 2018-09-24 13:22 UTC (permalink / raw)
To: Halil Pasic, Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On 24.09.2018 14:16, Halil Pasic wrote:
>
> On 09/24/2018 01:36 PM, Cornelia Huck wrote:
>> On Wed, 12 Sep 2018 15:43:03 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>
>>> Let's call PAPQ(ZAPQ) to zeroize a queue for each queue configured
>>> for a mediated matrix device when it is released.
>>>
>>> Zeroizing a queue resets the queue, clears all pending
>>> messages for the queue entries and disables adapter interruptions
>>> associated with the queue.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>> ---
>>> drivers/s390/crypto/vfio_ap_ops.c | 44 +++++++++++++++++++++++++++++++++++++
>>> 1 files changed, 44 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
>>> index f8b276a..48b1b78 100644
>>> --- a/drivers/s390/crypto/vfio_ap_ops.c
>>> +++ b/drivers/s390/crypto/vfio_ap_ops.c
>>> @@ -829,6 +829,49 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
>>> return NOTIFY_OK;
>>> }
>>>
>>> +static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
>>> + unsigned int retry)
>>> +{
>>> + struct ap_queue_status status;
>>> +
>>> + do {
>>> + status = ap_zapq(AP_MKQID(apid, apqi));
>>> + switch (status.response_code) {
>>> + case AP_RESPONSE_NORMAL:
>>> + return 0;
>>> + case AP_RESPONSE_RESET_IN_PROGRESS:
>>> + case AP_RESPONSE_BUSY:
>>> + msleep(20);
>>> + break;
>>> + default:
>>> + /* things are really broken, give up */
>>> + return -EIO;
>>> + }
>>> + } while (retry--);
>>> +
>>> + return -EBUSY;
>> So, this function may either return 0, -EIO (things are really broken),
>> or -EBUSY (still busy after multiple tries)...
>>
>>> +}
>>> +
>>> +static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
>>> +{
>>> + int ret;
>>> + int rc = 0;
>>> + unsigned long apid, apqi;
>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>> +
>>> + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
>>> + matrix_mdev->matrix.apm_max + 1) {
>>> + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
>>> + matrix_mdev->matrix.aqm_max + 1) {
>>> + ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
>>> + if (ret)
>>> + rc = ret;
>> ...and here, we return the last error of any of the resets. Two
>> questions:
>>
>> - Does it make sense to continue if we get -EIO? IOW, does "really
>> broken" only refer to a certain tuple and other tuples still can/need
>> to be reset?
> I think it does make sense to continue, because IMHO "things are really
> broken" is an overstatement (I mean the APQN invalid case). One could
> argue would skipping the current card (adapter) be justified or not.
>
> IMHO the current code is good enough for the first shot, and we can think
> about fine-tuning it later.
Absolutely. The -EIO case is reached for example when the APQN
is 'deconfigured' which means the crypto adapter is logically unplugged.
So the -EIO case should NOT lead to some fatal actions like panic()
or cause a KVM guest to shut down or so.
>> - Is the return code useful in any way, as we don't know which tuple it
>> refers to?
>>
> Well, good question. It conveys that the operation can 'fail'. AFAIR -EBUSY
> is mostly fine given what the architecture say if we are satisfied with just
> reset. And the cases behind -EIO might actually be OK too in the same sense.
> My guess is, that based on the return value client code can tell if we have
> zeroize for all queues or basically just reset (like rapq). We could log that
> to some debug facility or whatever -- I guess, but at the moment we don't care.
>
> In the end I think the code is good enough as is, and if we want we can
> improve on it later.
>
> Regards,
> Halil
>
>
>>> + }
>>> + }
>>> +
>>> + return rc;
>>> +}
>>> +
>>> static int vfio_ap_mdev_open(struct mdev_device *mdev)
>>> {
>>> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>> @@ -859,6 +902,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
>>> if (matrix_mdev->kvm)
>>> kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>>>
>>> + vfio_ap_mdev_reset_queues(mdev);
>>> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>>> &matrix_mdev->group_notifier);
>>> matrix_mdev->kvm = NULL;
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues
2018-09-24 13:22 ` Harald Freudenberger
@ 2018-09-24 16:42 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:42 UTC (permalink / raw)
To: Harald Freudenberger, Halil Pasic, Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/24/2018 09:22 AM, Harald Freudenberger wrote:
> On 24.09.2018 14:16, Halil Pasic wrote:
>>
>> On 09/24/2018 01:36 PM, Cornelia Huck wrote:
(...)
>>> ...and here, we return the last error of any of the resets. Two
>>> questions:
>>>
>>> - Does it make sense to continue if we get -EIO? IOW, does "really
>>> broken" only refer to a certain tuple and other tuples still can/need
>>> to be reset?
>> I think it does make sense to continue, because IMHO "things are really
>> broken" is an overstatement (I mean the APQN invalid case). One could
>> argue would skipping the current card (adapter) be justified or not.
>>
>> IMHO the current code is good enough for the first shot, and we can think
>> about fine-tuning it later.
> Absolutely. The -EIO case is reached for example when the APQN
> is 'deconfigured' which means the crypto adapter is logically unplugged.
> So the -EIO case should NOT lead to some fatal actions like panic()
> or cause a KVM guest to shut down or so.
>>> - Is the return code useful in any way, as we don't know which tuple it
>>> refers to?
>>>
>> Well, good question. It conveys that the operation can 'fail'. AFAIR -EBUSY
>> is mostly fine given what the architecture say if we are satisfied with just
>> reset. And the cases behind -EIO might actually be OK too in the same sense.
>> My guess is, that based on the return value client code can tell if we have
>> zeroize for all queues or basically just reset (like rapq). We could log that
>> to some debug facility or whatever -- I guess, but at the moment we don't care.
>>
>> In the end I think the code is good enough as is, and if we want we can
>> improve on it later.
>>
>> Regards,
>> Halil
>>
I'll note that in v7 a message was logged to indicate for which APQN the
error occurred, but I was asked to remove the printk log messsages. I
agree with Halil and Harald confirmed that the code is probably okay as
it stands. I can definitely see enhancing all of AP virtualization down
the road with some type of debug logging.
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 14/26] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (12 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 13/26] s390: vfio-ap: zeroize the AP queues Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-24 11:43 ` Cornelia Huck
2018-09-12 19:43 ` [PATCH v10 15/26] KVM: s390: Clear Crypto Control Block when using vSIE Tony Krowiak
` (13 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Implements the VFIO_DEVICE_RESET ioctl. This ioctl zeroizes
all of the AP queues assigned to the guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 48b1b78..d1b326b 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -922,7 +922,7 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg)
if (info.argsz < minsz)
return -EINVAL;
- info.flags = VFIO_DEVICE_FLAGS_AP;
+ info.flags = VFIO_DEVICE_FLAGS_AP | VFIO_DEVICE_FLAGS_RESET;
info.num_regions = 0;
info.num_irqs = 0;
@@ -938,6 +938,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
case VFIO_DEVICE_GET_INFO:
ret = vfio_ap_mdev_get_device_info(arg);
break;
+ case VFIO_DEVICE_RESET:
+ ret = vfio_ap_mdev_reset_queues(mdev);
+ break;
default:
ret = -EOPNOTSUPP;
break;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 14/26] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
2018-09-12 19:43 ` [PATCH v10 14/26] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl Tony Krowiak
@ 2018-09-24 11:43 ` Cornelia Huck
0 siblings, 0 replies; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:43 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:43:04 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Implements the VFIO_DEVICE_RESET ioctl. This ioctl zeroizes
> all of the AP queues assigned to the guest.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
> Tested-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 5 ++++-
> 1 files changed, 4 insertions(+), 1 deletions(-)
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 15/26] KVM: s390: Clear Crypto Control Block when using vSIE
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (13 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 14/26] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 16/26] KVM: s390: vsie: Do the CRYCB validation first Tony Krowiak
` (12 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When we clear the Crypto Control Block (CRYCB) used by a guest
level 2, the vSIE shadow CRYCB for guest level 3 must be updated
before the guest uses it.
We achieve this by using the KVM_REQ_VSIE_RESTART synchronous
request for each vCPU belonging to the guest to force the reload
of the shadow CRYCB before rerunning the guest level 3.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/kvm-s390.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index ac4c93f..2cdd980 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2043,6 +2043,8 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm)
memset(&kvm->arch.crypto.crycb->apcb1, 0,
sizeof(kvm->arch.crypto.crycb->apcb1));
+ /* recreate the shadow crycb for each vcpu */
+ kvm_s390_sync_request_broadcast(kvm, KVM_REQ_VSIE_RESTART);
kvm_s390_vcpu_unblock_all(kvm);
mutex_unlock(&kvm->lock);
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 16/26] KVM: s390: vsie: Do the CRYCB validation first
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (14 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 15/26] KVM: s390: Clear Crypto Control Block when using vSIE Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 17/26] KVM: s390: vsie: Make use of CRYCB FORMAT2 clear Tony Krowiak
` (11 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
We need to handle the validity checks for the crycb, no matter what the
settings for the keywrappings are. So lets move the keywrapping checks
after we have done the validy checks.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
---
arch/s390/kvm/vsie.c | 11 ++++++-----
1 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 12b9707..38ea5da 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -161,17 +161,18 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
/* format-1 is supported with message-security-assist extension 3 */
if (!test_kvm_facility(vcpu->kvm, 76))
return 0;
- /* we may only allow it if enabled for guest 2 */
- ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &
- (ECB3_AES | ECB3_DEA);
- if (!ecb3_flags)
- return 0;
if ((crycb_addr & PAGE_MASK) != ((crycb_addr + 128) & PAGE_MASK))
return set_validity_icpt(scb_s, 0x003CU);
else if (!crycb_addr)
return set_validity_icpt(scb_s, 0x0039U);
+ /* we may only allow it if enabled for guest 2 */
+ ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &
+ (ECB3_AES | ECB3_DEA);
+ if (!ecb3_flags)
+ return 0;
+
/* copy only the wrapping keys */
if (read_guest_real(vcpu, crycb_addr + 72,
vsie_page->crycb.dea_wrapping_key_mask, 56))
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 17/26] KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (15 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 16/26] KVM: s390: vsie: Do the CRYCB validation first Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 18/26] KVM: s390: vsie: Allow CRYCB FORMAT-2 Tony Krowiak
` (10 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
The comment preceding the shadow_crycb function is
misleading, we effectively accept FORMAT2 CRYCB in the
guest.
When using FORMAT2 in the host we do not need to or with
FORMAT1.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
---
arch/s390/kvm/vsie.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 38ea5da..e0e6fbf 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -140,7 +140,8 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
* Create a shadow copy of the crycb block and setup key wrapping, if
* requested for guest 3 and enabled for guest 2.
*
- * We only accept format-1 (no AP in g2), but convert it into format-2
+ * We accept format-1 or format-2, but we treat it as a format-1 (no AP in g2),
+ * and we convert it into format-2 in the shadow CRYCB.
* There is nothing to do for format-0.
*
* Returns: - 0 if shadowed or nothing to do
@@ -179,8 +180,7 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return set_validity_icpt(scb_s, 0x0035U);
scb_s->ecb3 |= ecb3_flags;
- scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT1 |
- CRYCB_FORMAT2;
+ scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT2;
/* xor both blocks in one run */
b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 18/26] KVM: s390: vsie: Allow CRYCB FORMAT-2
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (16 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 17/26] KVM: s390: vsie: Make use of CRYCB FORMAT2 clear Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 19/26] KVM: s390: vsie: allow CRYCB FORMAT-1 Tony Krowiak
` (9 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the guest and the host both use CRYCB FORMAT-2,
we copy the guest's FORMAT-1 APCB to a FORMAT-1
shadow APCB.
This patch also cleans up the shadow_crycb() function.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 114 +++++++++++++++++++++++++++++++++++++++++++++----
1 files changed, 104 insertions(+), 10 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index e0e6fbf..3a93278 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -136,14 +136,81 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return 0;
}
-/*
+/**
+ * setup_apcb11 - Copy the FORMAT1 APCB from the guest to the shadow CRYCB
+ * @vcpu: pointer to the virtual CPU
+ * @apcb_s: pointer to start of apcb in the shadow crycb
+ * @apcb_o: pointer to start of original guest apcb
+ * @apcb_h: pointer to start of apcb in the host
+ *
+ * Returns 0 and -EFAULT on error reading guest apcb
+ */
+static int setup_apcb11(struct kvm_vcpu *vcpu, unsigned long *apcb_s,
+ unsigned long apcb_o,
+ unsigned long *apcb_h)
+{
+ if (read_guest_real(vcpu, apcb_o, apcb_s,
+ sizeof(struct kvm_s390_apcb1)))
+ return -EFAULT;
+
+ bitmap_and(apcb_s, apcb_s, apcb_h, sizeof(struct kvm_s390_apcb1));
+
+ return 0;
+}
+
+/**
+ * setup_apcb - Create a shadow copy of the apcb.
+ * @vcpu: pointer to the virtual CPU
+ * @crycb_s: pointer to shadow crycb
+ * @crycb_o: pointer to original guest crycb
+ * @crycb_h: pointer to the host crycb
+ * @fmt_o: format of the original guest crycb.
+ * @fmt_h: format of the host crycb.
+ *
+ * Checks the compatibility between the guest and host crycb and calls the
+ * appropriate copy function.
+ *
+ * Return 0 or an error number if the guest and host crycb are incompatible.
+ */
+static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
+ const u32 crycb_o,
+ struct kvm_s390_crypto_cb *crycb_h,
+ int fmt_o, int fmt_h)
+{
+ struct kvm_s390_crypto_cb *crycb;
+
+ crycb = (struct kvm_s390_crypto_cb *) (unsigned long)crycb_o;
+
+ switch (fmt_o) {
+ case CRYCB_FORMAT2:
+ if ((crycb_o & PAGE_MASK) != ((crycb_o + 256) & PAGE_MASK))
+ return -EACCES;
+ if (fmt_h != CRYCB_FORMAT2)
+ return -EINVAL;
+ return setup_apcb11(vcpu, (unsigned long *)&crycb_s->apcb1,
+ (unsigned long) &crycb->apcb1,
+ (unsigned long *)&crycb_h->apcb1);
+ }
+ return -EINVAL;
+}
+
+/**
+ * shadow_crycb - Create a shadow copy of the crycb block
+ * @vcpu: a pointer to the virtual CPU
+ * @vsie_page: a pointer to internal date used for the vSIE
+ *
* Create a shadow copy of the crycb block and setup key wrapping, if
* requested for guest 3 and enabled for guest 2.
*
- * We accept format-1 or format-2, but we treat it as a format-1 (no AP in g2),
- * and we convert it into format-2 in the shadow CRYCB.
+ * We accept format-1 or format-2, but we convert format-1 into format-2
+ * in the shadow CRYCB.
+ * Using format-2 enables the firmware to choose the right format when
+ * scheduling the SIE.
* There is nothing to do for format-0.
*
+ * This function centralize the issuing of set_validity_icpt() for all
+ * the subfunctions working on the crycb.
+ *
* Returns: - 0 if shadowed or nothing to do
* - > 0 if control has to be given to guest 2
*/
@@ -155,24 +222,42 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
const u32 crycb_addr = crycbd_o & 0x7ffffff8U;
unsigned long *b1, *b2;
u8 ecb3_flags;
+ int apie_h;
+ int key_msk = test_kvm_facility(vcpu->kvm, 76);
+ int fmt_o = crycbd_o & CRYCB_FORMAT_MASK;
+ int fmt_h = vcpu->arch.sie_block->crycbd & CRYCB_FORMAT_MASK;
+ int ret = 0;
scb_s->crycbd = 0;
if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
return 0;
- /* format-1 is supported with message-security-assist extension 3 */
- if (!test_kvm_facility(vcpu->kvm, 76))
+
+ apie_h = vcpu->arch.sie_block->eca & ECA_APIE;
+ if (!apie_h && !key_msk)
return 0;
- if ((crycb_addr & PAGE_MASK) != ((crycb_addr + 128) & PAGE_MASK))
- return set_validity_icpt(scb_s, 0x003CU);
- else if (!crycb_addr)
+ if (!crycb_addr)
return set_validity_icpt(scb_s, 0x0039U);
+ if (fmt_o == CRYCB_FORMAT1)
+ if ((crycb_addr & PAGE_MASK) !=
+ ((crycb_addr + 128) & PAGE_MASK))
+ return set_validity_icpt(scb_s, 0x003CU);
+
+ if (apie_h && (scb_o->eca & ECA_APIE)) {
+ ret = setup_apcb(vcpu, &vsie_page->crycb, crycb_addr,
+ vcpu->kvm->arch.crypto.crycb,
+ fmt_o, fmt_h);
+ if (ret)
+ goto end;
+ scb_s->eca |= scb_o->eca & ECA_APIE;
+ }
+
/* we may only allow it if enabled for guest 2 */
ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &
(ECB3_AES | ECB3_DEA);
if (!ecb3_flags)
- return 0;
+ goto end;
/* copy only the wrapping keys */
if (read_guest_real(vcpu, crycb_addr + 72,
@@ -180,7 +265,6 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return set_validity_icpt(scb_s, 0x0035U);
scb_s->ecb3 |= ecb3_flags;
- scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT2;
/* xor both blocks in one run */
b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
@@ -188,6 +272,16 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
vcpu->kvm->arch.crypto.crycb->dea_wrapping_key_mask;
/* as 56%8 == 0, bitmap_xor won't overwrite any data */
bitmap_xor(b1, b1, b2, BITS_PER_BYTE * 56);
+end:
+ switch (ret) {
+ case -EINVAL:
+ return set_validity_icpt(scb_s, 0x0020U);
+ case -EFAULT:
+ return set_validity_icpt(scb_s, 0x0035U);
+ case -EACCES:
+ return set_validity_icpt(scb_s, 0x003CU);
+ }
+ scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT2;
return 0;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 19/26] KVM: s390: vsie: allow CRYCB FORMAT-1
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (17 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 18/26] KVM: s390: vsie: Allow CRYCB FORMAT-2 Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 20/26] KVM: s390: vsie: allow CRYCB FORMAT-0 Tony Krowiak
` (8 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the host and guest both use a FORMAT-1 CRYCB, we copy
the guest's FORMAT-0 APCB to a shadow CRYCB for use by
vSIE.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 19 +++++++++++++++++++
1 files changed, 19 insertions(+), 0 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 3a93278..0a26814 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -136,6 +136,19 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return 0;
}
+/* Copy to APCB FORMAT0 from APCB FORMAT0 */
+static int setup_apcb00(struct kvm_vcpu *vcpu, unsigned long *apcb_s,
+ unsigned long apcb_o, unsigned long *apcb_h)
+{
+ if (read_guest_real(vcpu, apcb_o, apcb_s,
+ sizeof(struct kvm_s390_apcb0)))
+ return -EFAULT;
+
+ bitmap_and(apcb_s, apcb_s, apcb_h, sizeof(struct kvm_s390_apcb0));
+
+ return 0;
+}
+
/**
* setup_apcb11 - Copy the FORMAT1 APCB from the guest to the shadow CRYCB
* @vcpu: pointer to the virtual CPU
@@ -190,6 +203,12 @@ static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
return setup_apcb11(vcpu, (unsigned long *)&crycb_s->apcb1,
(unsigned long) &crycb->apcb1,
(unsigned long *)&crycb_h->apcb1);
+ case CRYCB_FORMAT1:
+ if (fmt_h != CRYCB_FORMAT1)
+ return -EINVAL;
+ return setup_apcb00(vcpu, (unsigned long *) &crycb_s->apcb0,
+ (unsigned long) &crycb->apcb0,
+ (unsigned long *) &crycb_h->apcb0);
}
return -EINVAL;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 20/26] KVM: s390: vsie: allow CRYCB FORMAT-0
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (18 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 19/26] KVM: s390: vsie: allow CRYCB FORMAT-1 Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 21/26] KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1 Tony Krowiak
` (7 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the host and the guest both use a FORMAT-0 CRYCB,
we copy the guest's FORMAT-0 APCB to a shadow CRYCB
for use by vSIE.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 20 +++++++++++++++++---
1 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 0a26814..67a5536 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -136,7 +136,15 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return 0;
}
-/* Copy to APCB FORMAT0 from APCB FORMAT0 */
+/**
+ * setup_apcb00 - Copy to APCB FORMAT0 from APCB FORMAT0
+ * @vcpu: pointer to the virtual CPU
+ * @apcb_s: pointer to start of apcb in the shadow crycb
+ * @apcb_o: pointer to start of original apcb in the guest2
+ * @apcb_h: pointer to start of apcb in the guest1
+ *
+ * Returns 0 and -EFAULT on error reading guest apcb
+ */
static int setup_apcb00(struct kvm_vcpu *vcpu, unsigned long *apcb_s,
unsigned long apcb_o, unsigned long *apcb_h)
{
@@ -209,6 +217,14 @@ static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
return setup_apcb00(vcpu, (unsigned long *) &crycb_s->apcb0,
(unsigned long) &crycb->apcb0,
(unsigned long *) &crycb_h->apcb0);
+ case CRYCB_FORMAT0:
+ if ((crycb_o & PAGE_MASK) != ((crycb_o + 32) & PAGE_MASK))
+ return -EACCES;
+ if (fmt_h != CRYCB_FORMAT0)
+ return -EINVAL;
+ return setup_apcb00(vcpu, (unsigned long *) &crycb_s->apcb0,
+ (unsigned long) &crycb->apcb0,
+ (unsigned long *) &crycb_h->apcb0);
}
return -EINVAL;
}
@@ -248,8 +264,6 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
int ret = 0;
scb_s->crycbd = 0;
- if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
- return 0;
apie_h = vcpu->arch.sie_block->eca & ECA_APIE;
if (!apie_h && !key_msk)
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 21/26] KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (19 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 20/26] KVM: s390: vsie: allow CRYCB FORMAT-0 Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 22/26] KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2 Tony Krowiak
` (6 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the guest schedules a SIE with a FORMAT-0 CRYCB,
we are able to schedule it in the host with a FORMAT-1
CRYCB if the host uses FORMAT-1 or FORMAT-0.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 14 ++++++++++----
1 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 67a5536..90ae310 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -220,11 +220,17 @@ static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
case CRYCB_FORMAT0:
if ((crycb_o & PAGE_MASK) != ((crycb_o + 32) & PAGE_MASK))
return -EACCES;
- if (fmt_h != CRYCB_FORMAT0)
+
+ switch (fmt_h) {
+ case CRYCB_FORMAT2:
return -EINVAL;
- return setup_apcb00(vcpu, (unsigned long *) &crycb_s->apcb0,
- (unsigned long) &crycb->apcb0,
- (unsigned long *) &crycb_h->apcb0);
+ case CRYCB_FORMAT1:
+ case CRYCB_FORMAT0:
+ return setup_apcb00(vcpu,
+ (unsigned long *) &crycb_s->apcb0,
+ (unsigned long) &crycb->apcb0,
+ (unsigned long *) &crycb_h->apcb0);
+ }
}
return -EINVAL;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 22/26] KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (20 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 21/26] KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1 Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 23/26] KVM: s390: vsie: allow guest FORMAT-0 " Tony Krowiak
` (5 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the guest schedules a SIE with a CRYCB FORMAT-1 CRYCB,
we are able to schedule it in the host with a FORMAT-2 CRYCB
if the host uses FORMAT-2.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 33 ++++++++++++++++++++++++++++-----
1 files changed, 28 insertions(+), 5 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 90ae310..ca64663 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -135,6 +135,22 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
atomic_set(&scb_s->cpuflags, newflags);
return 0;
}
+/* Copy to APCB FORMAT1 from APCB FORMAT0 */
+static int setup_apcb10(struct kvm_vcpu *vcpu, struct kvm_s390_apcb1 *apcb_s,
+ unsigned long apcb_o, struct kvm_s390_apcb1 *apcb_h)
+{
+ struct kvm_s390_apcb0 tmp;
+
+ if (read_guest_real(vcpu, apcb_o, &tmp, sizeof(struct kvm_s390_apcb0)))
+ return -EFAULT;
+
+ apcb_s->apm[0] = apcb_h->apm[0] & tmp.apm[0];
+ apcb_s->aqm[0] = apcb_h->aqm[0] & tmp.aqm[0] & 0xffff000000000000UL;
+ apcb_s->adm[0] = apcb_h->adm[0] & tmp.adm[0] & 0xffff000000000000UL;
+
+ return 0;
+
+}
/**
* setup_apcb00 - Copy to APCB FORMAT0 from APCB FORMAT0
@@ -212,11 +228,18 @@ static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
(unsigned long) &crycb->apcb1,
(unsigned long *)&crycb_h->apcb1);
case CRYCB_FORMAT1:
- if (fmt_h != CRYCB_FORMAT1)
- return -EINVAL;
- return setup_apcb00(vcpu, (unsigned long *) &crycb_s->apcb0,
- (unsigned long) &crycb->apcb0,
- (unsigned long *) &crycb_h->apcb0);
+ switch (fmt_h) {
+ case CRYCB_FORMAT2:
+ return setup_apcb10(vcpu, &crycb_s->apcb1,
+ (unsigned long) &crycb->apcb0,
+ &crycb_h->apcb1);
+ case CRYCB_FORMAT1:
+ return setup_apcb00(vcpu,
+ (unsigned long *) &crycb_s->apcb0,
+ (unsigned long) &crycb->apcb0,
+ (unsigned long *) &crycb_h->apcb0);
+ }
+ break;
case CRYCB_FORMAT0:
if ((crycb_o & PAGE_MASK) != ((crycb_o + 32) & PAGE_MASK))
return -EACCES;
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 23/26] KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (21 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 22/26] KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2 Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation Tony Krowiak
` (4 subsequent siblings)
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Pierre Morel, Tony Krowiak
From: Pierre Morel <pmorel@linux.ibm.com>
When the guest schedules a SIE with a FORMAT-0 CRYCB,
we are able to schedule it in the host with a FORMAT-2
CRYCB if the host uses FORMAT-2
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/kvm/vsie.c | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index ca64663..9117891 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -246,7 +246,9 @@ static int setup_apcb(struct kvm_vcpu *vcpu, struct kvm_s390_crypto_cb *crycb_s,
switch (fmt_h) {
case CRYCB_FORMAT2:
- return -EINVAL;
+ return setup_apcb10(vcpu, &crycb_s->apcb1,
+ (unsigned long) &crycb->apcb0,
+ &crycb_h->apcb1);
case CRYCB_FORMAT1:
case CRYCB_FORMAT0:
return setup_apcb00(vcpu,
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (22 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 23/26] KVM: s390: vsie: allow guest FORMAT-0 " Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-17 8:51 ` David Hildenbrand
2018-09-12 19:43 ` [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization Tony Krowiak
` (3 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
to enable or disable AP instruction interpretation from userspace
via the KVM_SET_DEVICE_ATTR ioctl:
* The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
interpretation of AP instructions executed on the guest.
* The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
interpretation of AP instructions executed on the guest. In this
case the instructions will be intercepted and pass through to
the guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/include/uapi/asm/kvm.h | 2 ++
arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
3 files changed, 26 insertions(+), 4 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index b32bd1b..36d3531 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -719,6 +719,7 @@ struct kvm_s390_crypto {
__u32 crycbd;
__u8 aes_kw;
__u8 dea_kw;
+ __u8 apie;
};
#define APCB0_MASK_SIZE 1
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index 8c23afc..a8dbd90 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
#define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
#define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
+#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
+#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
/* kvm attributes for migration mode */
#define KVM_S390_VM_MIGRATION_STOP 0
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 2cdd980..286c2e0 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
{
- if (!test_kvm_facility(kvm, 76))
- return -EINVAL;
-
mutex_lock(&kvm->lock);
switch (attr->attr) {
case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
+ if (!test_kvm_facility(kvm, 76))
+ return -EINVAL;
get_random_bytes(
kvm->arch.crypto.crycb->aes_wrapping_key_mask,
sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
@@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
break;
case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
+ if (!test_kvm_facility(kvm, 76))
+ return -EINVAL;
get_random_bytes(
kvm->arch.crypto.crycb->dea_wrapping_key_mask,
sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
@@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
break;
case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
+ if (!test_kvm_facility(kvm, 76))
+ return -EINVAL;
kvm->arch.crypto.aes_kw = 0;
memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
break;
case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
+ if (!test_kvm_facility(kvm, 76))
+ return -EINVAL;
kvm->arch.crypto.dea_kw = 0;
memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
break;
+ case KVM_S390_VM_CRYPTO_ENABLE_APIE:
+ if (!ap_instructions_available()) {
+ mutex_unlock(&kvm->lock);
+ return -EOPNOTSUPP;
+ }
+ kvm->arch.crypto.apie = 1;
+ break;
+ case KVM_S390_VM_CRYPTO_DISABLE_APIE:
+ kvm->arch.crypto.apie = 0;
+ break;
default:
mutex_unlock(&kvm->lock);
return -ENXIO;
@@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
+ case KVM_S390_VM_CRYPTO_ENABLE_APIE:
+ case KVM_S390_VM_CRYPTO_DISABLE_APIE:
ret = 0;
break;
default:
@@ -2062,6 +2079,7 @@ static u64 kvm_s390_get_initial_cpuid(void)
static void kvm_s390_crypto_init(struct kvm *kvm)
{
kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
+ kvm->arch.crypto.apie = 0;
kvm_s390_set_crycb_format(kvm);
if (!test_kvm_facility(kvm, 76))
@@ -2602,8 +2620,9 @@ static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
+ vcpu->arch.sie_block->eca &= ~ECA_APIE;
- if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
+ if (vcpu->kvm->arch.crypto.apie)
vcpu->arch.sie_block->eca |= ECA_APIE;
/* Set up protected key support */
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-12 19:43 ` [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation Tony Krowiak
@ 2018-09-17 8:51 ` David Hildenbrand
2018-09-21 23:40 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-17 8:51 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja, Tony Krowiak
Am 12.09.18 um 21:43 schrieb Tony Krowiak:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
> to enable or disable AP instruction interpretation from userspace
> via the KVM_SET_DEVICE_ATTR ioctl:
>
> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
> interpretation of AP instructions executed on the guest.
>
> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
> interpretation of AP instructions executed on the guest. In this
> case the instructions will be intercepted and pass through to
> the guest.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> ---
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/include/uapi/asm/kvm.h | 2 ++
> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
> 3 files changed, 26 insertions(+), 4 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index b32bd1b..36d3531 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
> __u32 crycbd;
> __u8 aes_kw;
> __u8 dea_kw;
> + __u8 apie;
> };
>
> #define APCB0_MASK_SIZE 1
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index 8c23afc..a8dbd90 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>
> /* kvm attributes for migration mode */
> #define KVM_S390_VM_MIGRATION_STOP 0
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 2cdd980..286c2e0 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>
> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
> {
> - if (!test_kvm_facility(kvm, 76))
> - return -EINVAL;
> -
> mutex_lock(&kvm->lock);
> switch (attr->attr) {
> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
> + if (!test_kvm_facility(kvm, 76))
> + return -EINVAL;
> get_random_bytes(
> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
> break;
> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
> + if (!test_kvm_facility(kvm, 76))
> + return -EINVAL;
> get_random_bytes(
> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
> break;
> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
> + if (!test_kvm_facility(kvm, 76))
> + return -EINVAL;
> kvm->arch.crypto.aes_kw = 0;
> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
> break;
> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
> + if (!test_kvm_facility(kvm, 76))
> + return -EINVAL;
> kvm->arch.crypto.dea_kw = 0;
> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
> break;
> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
> + if (!ap_instructions_available()) {
> + mutex_unlock(&kvm->lock);
> + return -EOPNOTSUPP;
> + }
> + kvm->arch.crypto.apie = 1;
> + break;
> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> + kvm->arch.crypto.apie = 0;
> + break;
> default:
> mutex_unlock(&kvm->lock);
> return -ENXIO;
> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
As also replied to the QEMU series, could we indicate
KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
(never checked, we only care about apie).
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-17 8:51 ` David Hildenbrand
@ 2018-09-21 23:40 ` Tony Krowiak
2018-09-24 11:23 ` David Hildenbrand
0 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-21 23:40 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/17/2018 04:51 AM, David Hildenbrand wrote:
> Am 12.09.18 um 21:43 schrieb Tony Krowiak:
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
>> to enable or disable AP instruction interpretation from userspace
>> via the KVM_SET_DEVICE_ATTR ioctl:
>>
>> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
>> interpretation of AP instructions executed on the guest.
>>
>> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
>> interpretation of AP instructions executed on the guest. In this
>> case the instructions will be intercepted and pass through to
>> the guest.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> ---
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/include/uapi/asm/kvm.h | 2 ++
>> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
>> 3 files changed, 26 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index b32bd1b..36d3531 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
>> __u32 crycbd;
>> __u8 aes_kw;
>> __u8 dea_kw;
>> + __u8 apie;
>> };
>>
>> #define APCB0_MASK_SIZE 1
>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>> index 8c23afc..a8dbd90 100644
>> --- a/arch/s390/include/uapi/asm/kvm.h
>> +++ b/arch/s390/include/uapi/asm/kvm.h
>> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
>> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
>> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
>> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
>> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
>> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>>
>> /* kvm attributes for migration mode */
>> #define KVM_S390_VM_MIGRATION_STOP 0
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 2cdd980..286c2e0 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>
>> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>> {
>> - if (!test_kvm_facility(kvm, 76))
>> - return -EINVAL;
>> -
>> mutex_lock(&kvm->lock);
>> switch (attr->attr) {
>> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
>> + if (!test_kvm_facility(kvm, 76))
>> + return -EINVAL;
>> get_random_bytes(
>> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
>> break;
>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>> + if (!test_kvm_facility(kvm, 76))
>> + return -EINVAL;
>> get_random_bytes(
>> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
>> break;
>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>> + if (!test_kvm_facility(kvm, 76))
>> + return -EINVAL;
>> kvm->arch.crypto.aes_kw = 0;
>> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
>> break;
>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>> + if (!test_kvm_facility(kvm, 76))
>> + return -EINVAL;
>> kvm->arch.crypto.dea_kw = 0;
>> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
>> break;
>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>> + if (!ap_instructions_available()) {
>> + mutex_unlock(&kvm->lock);
>> + return -EOPNOTSUPP;
>> + }
>> + kvm->arch.crypto.apie = 1;
>> + break;
>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>> + kvm->arch.crypto.apie = 0;
>> + break;
>> default:
>> mutex_unlock(&kvm->lock);
>> return -ENXIO;
>> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>
> As also replied to the QEMU series, could we indicate
> KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
> KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
> so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
>
> KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
> (never checked, we only care about apie).
After much discussion with Halil and a few exchanges with you, we
decided to go ahead and accept your suggestion to get rid of
KVM_S390_VM_CPU_FEAT and keep the VM device attributes to enable/disable
apie.
To that end, I responded to patches 03/26, 11/26 and 25/26 with fixup!
patches that show the KVM/kernel changes that will be necessary to get
rid of KVM_S390_VM_CPU_FEAT and use apie to control ECA.28. I did that
to generate discussion in v10 rather than waiting until v11 for
comments. I make no guarantees that those fixup! patches will
successfully apply should you have a v10 branch generated from this
patch series you want to update.
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-21 23:40 ` Tony Krowiak
@ 2018-09-24 11:23 ` David Hildenbrand
2018-09-24 16:25 ` Tony Krowiak
0 siblings, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 11:23 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 22/09/2018 01:40, Tony Krowiak wrote:
> On 09/17/2018 04:51 AM, David Hildenbrand wrote:
>> Am 12.09.18 um 21:43 schrieb Tony Krowiak:
>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>
>>> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
>>> to enable or disable AP instruction interpretation from userspace
>>> via the KVM_SET_DEVICE_ATTR ioctl:
>>>
>>> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
>>> interpretation of AP instructions executed on the guest.
>>>
>>> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
>>> interpretation of AP instructions executed on the guest. In this
>>> case the instructions will be intercepted and pass through to
>>> the guest.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>>> ---
>>> arch/s390/include/asm/kvm_host.h | 1 +
>>> arch/s390/include/uapi/asm/kvm.h | 2 ++
>>> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
>>> 3 files changed, 26 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>> index b32bd1b..36d3531 100644
>>> --- a/arch/s390/include/asm/kvm_host.h
>>> +++ b/arch/s390/include/asm/kvm_host.h
>>> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
>>> __u32 crycbd;
>>> __u8 aes_kw;
>>> __u8 dea_kw;
>>> + __u8 apie;
>>> };
>>>
>>> #define APCB0_MASK_SIZE 1
>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>> index 8c23afc..a8dbd90 100644
>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
>>> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
>>> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
>>> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
>>> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
>>> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>>>
>>> /* kvm attributes for migration mode */
>>> #define KVM_S390_VM_MIGRATION_STOP 0
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index 2cdd980..286c2e0 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>>
>>> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>> {
>>> - if (!test_kvm_facility(kvm, 76))
>>> - return -EINVAL;
>>> -
>>> mutex_lock(&kvm->lock);
>>> switch (attr->attr) {
>>> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
>>> + if (!test_kvm_facility(kvm, 76))
>>> + return -EINVAL;
>>> get_random_bytes(
>>> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
>>> break;
>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>> + if (!test_kvm_facility(kvm, 76))
>>> + return -EINVAL;
>>> get_random_bytes(
>>> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
>>> break;
>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>> + if (!test_kvm_facility(kvm, 76))
>>> + return -EINVAL;
>>> kvm->arch.crypto.aes_kw = 0;
>>> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
>>> break;
>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>> + if (!test_kvm_facility(kvm, 76))
>>> + return -EINVAL;
>>> kvm->arch.crypto.dea_kw = 0;
>>> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
>>> break;
>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>> + if (!ap_instructions_available()) {
>>> + mutex_unlock(&kvm->lock);
>>> + return -EOPNOTSUPP;
>>> + }
>>> + kvm->arch.crypto.apie = 1;
>>> + break;
>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>> + kvm->arch.crypto.apie = 0;
>>> + break;
>>> default:
>>> mutex_unlock(&kvm->lock);
>>> return -ENXIO;
>>> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>
>> As also replied to the QEMU series, could we indicate
>> KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
>> KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
>> so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
>>
>> KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
>> (never checked, we only care about apie).
>
> After much discussion with Halil and a few exchanges with you, we
> decided to go ahead and accept your suggestion to get rid of
> KVM_S390_VM_CPU_FEAT and keep the VM device attributes to enable/disable
> apie.
>
> To that end, I responded to patches 03/26, 11/26 and 25/26 with fixup!
> patches that show the KVM/kernel changes that will be necessary to get
> rid of KVM_S390_VM_CPU_FEAT and use apie to control ECA.28. I did that
> to generate discussion in v10 rather than waiting until v11 for
> comments. I make no guarantees that those fixup! patches will
> successfully apply should you have a v10 branch generated from this
> patch series you want to update.
>
Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
only if supported by HW? (ap_instructions_available)
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 11:23 ` David Hildenbrand
@ 2018-09-24 16:25 ` Tony Krowiak
2018-09-24 18:42 ` Tony Krowiak
2018-09-24 18:46 ` David Hildenbrand
0 siblings, 2 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:25 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 07:23 AM, David Hildenbrand wrote:
> On 22/09/2018 01:40, Tony Krowiak wrote:
>> On 09/17/2018 04:51 AM, David Hildenbrand wrote:
>>> Am 12.09.18 um 21:43 schrieb Tony Krowiak:
>>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>
>>>> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
>>>> to enable or disable AP instruction interpretation from userspace
>>>> via the KVM_SET_DEVICE_ATTR ioctl:
>>>>
>>>> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
>>>> interpretation of AP instructions executed on the guest.
>>>>
>>>> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
>>>> interpretation of AP instructions executed on the guest. In this
>>>> case the instructions will be intercepted and pass through to
>>>> the guest.
>>>>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>>>> ---
>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>> arch/s390/include/uapi/asm/kvm.h | 2 ++
>>>> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
>>>> 3 files changed, 26 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>> index b32bd1b..36d3531 100644
>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
>>>> __u32 crycbd;
>>>> __u8 aes_kw;
>>>> __u8 dea_kw;
>>>> + __u8 apie;
>>>> };
>>>>
>>>> #define APCB0_MASK_SIZE 1
>>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>>> index 8c23afc..a8dbd90 100644
>>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>>> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
>>>> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
>>>> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
>>>> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
>>>> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
>>>> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>>>>
>>>> /* kvm attributes for migration mode */
>>>> #define KVM_S390_VM_MIGRATION_STOP 0
>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>> index 2cdd980..286c2e0 100644
>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>>>
>>>> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>> {
>>>> - if (!test_kvm_facility(kvm, 76))
>>>> - return -EINVAL;
>>>> -
>>>> mutex_lock(&kvm->lock);
>>>> switch (attr->attr) {
>>>> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
>>>> + if (!test_kvm_facility(kvm, 76))
>>>> + return -EINVAL;
>>>> get_random_bytes(
>>>> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
>>>> break;
>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>> + if (!test_kvm_facility(kvm, 76))
>>>> + return -EINVAL;
>>>> get_random_bytes(
>>>> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
>>>> break;
>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>> + if (!test_kvm_facility(kvm, 76))
>>>> + return -EINVAL;
>>>> kvm->arch.crypto.aes_kw = 0;
>>>> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
>>>> break;
>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>> + if (!test_kvm_facility(kvm, 76))
>>>> + return -EINVAL;
>>>> kvm->arch.crypto.dea_kw = 0;
>>>> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
>>>> break;
>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>> + if (!ap_instructions_available()) {
>>>> + mutex_unlock(&kvm->lock);
>>>> + return -EOPNOTSUPP;
>>>> + }
>>>> + kvm->arch.crypto.apie = 1;
>>>> + break;
>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>> + kvm->arch.crypto.apie = 0;
>>>> + break;
>>>> default:
>>>> mutex_unlock(&kvm->lock);
>>>> return -ENXIO;
>>>> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>
>>> As also replied to the QEMU series, could we indicate
>>> KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
>>> KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
>>> so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
>>>
>>> KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
>>> (never checked, we only care about apie).
>>
>> After much discussion with Halil and a few exchanges with you, we
>> decided to go ahead and accept your suggestion to get rid of
>> KVM_S390_VM_CPU_FEAT and keep the VM device attributes to enable/disable
>> apie.
>>
>> To that end, I responded to patches 03/26, 11/26 and 25/26 with fixup!
>> patches that show the KVM/kernel changes that will be necessary to get
>> rid of KVM_S390_VM_CPU_FEAT and use apie to control ECA.28. I did that
>> to generate discussion in v10 rather than waiting until v11 for
>> comments. I make no guarantees that those fixup! patches will
>> successfully apply should you have a v10 branch generated from this
>> patch series you want to update.
>>
>
> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
> only if supported by HW? (ap_instructions_available)
Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
supported by HW, I assume you are talking about
KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
ap_instructions_available() for disabling APIE because I didn't
think it necessary given that ECA.28 will be set to 0 (intercept) by
default, whether AP instructions are installed or not; so why not allow
disabling apie. I suppose from the perspective of consistency, since the
kvm_s390_vm_has_attr() function checks ap_instructions_available() for
both attributes, then it probably makes sense to add that check to
KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
regardless of whether AP instructions are available. It boils down to
whether APIE needs to be dynamically disabled at some point when it has
been enabled. The only case I can think of where that may be necessary
is if a guest is migrated to a system without AP instructions. I don't
think that can happen and may even be protected against precisely
because the VM attributes won't be available on the target system due to
no AP instructions. What say you?
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 16:25 ` Tony Krowiak
@ 2018-09-24 18:42 ` Tony Krowiak
2018-09-24 18:51 ` David Hildenbrand
2018-09-25 7:32 ` David Hildenbrand
2018-09-24 18:46 ` David Hildenbrand
1 sibling, 2 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 18:42 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 12:25 PM, Tony Krowiak wrote:
> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
(...)
>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>> only if supported by HW? (ap_instructions_available)
>
> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
> supported by HW, I assume you are talking about
> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
> ap_instructions_available() for disabling APIE because I didn't
> think it necessary given that ECA.28 will be set to 0 (intercept) by
> default, whether AP instructions are installed or not; so why not allow
> disabling apie. I suppose from the perspective of consistency, since the
> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
> both attributes, then it probably makes sense to add that check to
> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
> regardless of whether AP instructions are available. It boils down to
> whether APIE needs to be dynamically disabled at some point when it has
> been enabled. The only case I can think of where that may be necessary
> is if a guest is migrated to a system without AP instructions. I don't
> think that can happen and may even be protected against precisely
> because the VM attributes won't be available on the target system due to
> no AP instructions. What say you?
>
David,
I'm sorry, I misinterpreted what you were asking for. Check out the
fixup! patch below and let me know if that is what you are looking for.
If so, I will integrate that change and post v11 tomorrow (Tuesday 9/24).
-----------------------------------8<-----------------------------------
From: Tony Krowiak <akrowiak@linux.ibm.com>
Date: Mon, 24 Sep 2018 14:18:37 -0400
Subject: [FIXUP v10] fixup! KVM: s390: device attrs to enable/disable AP
interpretation
---
arch/s390/kvm/kvm-s390.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 6654bb1fc26a..a528558baa78 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -901,6 +901,10 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm,
struct kvm_device_attr *attr)
kvm->arch.crypto.apie = 1;
break;
case KVM_S390_VM_CRYPTO_DISABLE_APIE:
+ if (!ap_instructions_available()) {
+ mutex_unlock(&kvm->lock);
+ return -EOPNOTSUPP;
+ }
kvm->arch.crypto.apie = 0;
break;
default:
@@ -1509,9 +1513,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
struct kvm_device_attr *attr)
case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
+ ret = 0;
+ break;
case KVM_S390_VM_CRYPTO_ENABLE_APIE:
case KVM_S390_VM_CRYPTO_DISABLE_APIE:
- ret = 0;
+ ret = ap_instructions_available();
break;
default:
ret = -ENXIO;
@@ -2620,6 +2626,7 @@ static void kvm_s390_vcpu_crypto_setup(struct
kvm_vcpu *vcpu)
vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
+ vcpu->kvm->arch.crypto.apie &= ~ECA_APIE;
if (vcpu->kvm->arch.crypto.apie)
vcpu->arch.sie_block->eca |= ECA_APIE;
--
2.19.0.221.g150f307
>>
>
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 18:42 ` Tony Krowiak
@ 2018-09-24 18:51 ` David Hildenbrand
2018-09-25 13:24 ` Tony Krowiak
2018-09-25 7:32 ` David Hildenbrand
1 sibling, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 18:51 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 24/09/2018 20:42, Tony Krowiak wrote:
> On 09/24/2018 12:25 PM, Tony Krowiak wrote:
>> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>
> (...)
>
>>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>>> only if supported by HW? (ap_instructions_available)
>>
>> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
>> supported by HW, I assume you are talking about
>> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
>> ap_instructions_available() for disabling APIE because I didn't
>> think it necessary given that ECA.28 will be set to 0 (intercept) by
>> default, whether AP instructions are installed or not; so why not allow
>> disabling apie. I suppose from the perspective of consistency, since the
>> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
>> both attributes, then it probably makes sense to add that check to
>> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
>> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
>> regardless of whether AP instructions are available. It boils down to
>> whether APIE needs to be dynamically disabled at some point when it has
>> been enabled. The only case I can think of where that may be necessary
>> is if a guest is migrated to a system without AP instructions. I don't
>> think that can happen and may even be protected against precisely
>> because the VM attributes won't be available on the target system due to
>> no AP instructions. What say you?
>>
> David,
>
> I'm sorry, I misinterpreted what you were asking for. Check out the
> fixup! patch below and let me know if that is what you are looking for.
> If so, I will integrate that change and post v11 tomorrow (Tuesday 9/24).
>
> -----------------------------------8<-----------------------------------
>
> From: Tony Krowiak <akrowiak@linux.ibm.com>
> Date: Mon, 24 Sep 2018 14:18:37 -0400
> Subject: [FIXUP v10] fixup! KVM: s390: device attrs to enable/disable AP
> interpretation
>
> ---
> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6654bb1fc26a..a528558baa78 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -901,6 +901,10 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm,
> struct kvm_device_attr *attr)
> kvm->arch.crypto.apie = 1;
> break;
> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> + if (!ap_instructions_available()) {
> + mutex_unlock(&kvm->lock);
> + return -EOPNOTSUPP;
> + }
> kvm->arch.crypto.apie = 0;
> break;
> default:
> @@ -1509,9 +1513,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
> struct kvm_device_attr *attr)
> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
> + ret = 0;
> + break;
> case KVM_S390_VM_CRYPTO_ENABLE_APIE:
> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> - ret = 0;
> + ret = ap_instructions_available();
> break;
> default:
> ret = -ENXIO;
> @@ -2620,6 +2626,7 @@ static void kvm_s390_vcpu_crypto_setup(struct
> kvm_vcpu *vcpu)
>
> vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
> + vcpu->kvm->arch.crypto.apie &= ~ECA_APIE;
Did you mean to set vcpu->arch.sie_block->eca here?
>
> if (vcpu->kvm->arch.crypto.apie)
> vcpu->arch.sie_block->eca |= ECA_APIE;
>
Apart from that, just what I had in mind :)
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 18:51 ` David Hildenbrand
@ 2018-09-25 13:24 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-25 13:24 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 02:51 PM, David Hildenbrand wrote:
> On 24/09/2018 20:42, Tony Krowiak wrote:
>> On 09/24/2018 12:25 PM, Tony Krowiak wrote:
>>> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>>
>> (...)
>>
>>>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>>>> only if supported by HW? (ap_instructions_available)
>>>
>>> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
>>> supported by HW, I assume you are talking about
>>> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
>>> ap_instructions_available() for disabling APIE because I didn't
>>> think it necessary given that ECA.28 will be set to 0 (intercept) by
>>> default, whether AP instructions are installed or not; so why not allow
>>> disabling apie. I suppose from the perspective of consistency, since the
>>> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
>>> both attributes, then it probably makes sense to add that check to
>>> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
>>> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
>>> regardless of whether AP instructions are available. It boils down to
>>> whether APIE needs to be dynamically disabled at some point when it has
>>> been enabled. The only case I can think of where that may be necessary
>>> is if a guest is migrated to a system without AP instructions. I don't
>>> think that can happen and may even be protected against precisely
>>> because the VM attributes won't be available on the target system due to
>>> no AP instructions. What say you?
>>>
>> David,
>>
>> I'm sorry, I misinterpreted what you were asking for. Check out the
>> fixup! patch below and let me know if that is what you are looking for.
>> If so, I will integrate that change and post v11 tomorrow (Tuesday 9/24).
>>
>> -----------------------------------8<-----------------------------------
>>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>> Date: Mon, 24 Sep 2018 14:18:37 -0400
>> Subject: [FIXUP v10] fixup! KVM: s390: device attrs to enable/disable AP
>> interpretation
>>
>> ---
>> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
>> 1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 6654bb1fc26a..a528558baa78 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -901,6 +901,10 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm,
>> struct kvm_device_attr *attr)
>> kvm->arch.crypto.apie = 1;
>> break;
>> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>> + if (!ap_instructions_available()) {
>> + mutex_unlock(&kvm->lock);
>> + return -EOPNOTSUPP;
>> + }
>> kvm->arch.crypto.apie = 0;
>> break;
>> default:
>> @@ -1509,9 +1513,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
>> struct kvm_device_attr *attr)
>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>> + ret = 0;
>> + break;
>> case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>> - ret = 0;
>> + ret = ap_instructions_available();
>> break;
>> default:
>> ret = -ENXIO;
>> @@ -2620,6 +2626,7 @@ static void kvm_s390_vcpu_crypto_setup(struct
>> kvm_vcpu *vcpu)
>>
>> vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>> + vcpu->kvm->arch.crypto.apie &= ~ECA_APIE;
>
> Did you mean to set vcpu->arch.sie_block->eca here?
Yes!
>
>>
>> if (vcpu->kvm->arch.crypto.apie)
>> vcpu->arch.sie_block->eca |= ECA_APIE;
>>
>
> Apart from that, just what I had in mind :)
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 18:42 ` Tony Krowiak
2018-09-24 18:51 ` David Hildenbrand
@ 2018-09-25 7:32 ` David Hildenbrand
2018-09-25 13:26 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-25 7:32 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 24/09/2018 20:42, Tony Krowiak wrote:
> On 09/24/2018 12:25 PM, Tony Krowiak wrote:
>> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>
> (...)
>
>>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>>> only if supported by HW? (ap_instructions_available)
>>
>> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
>> supported by HW, I assume you are talking about
>> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
>> ap_instructions_available() for disabling APIE because I didn't
>> think it necessary given that ECA.28 will be set to 0 (intercept) by
>> default, whether AP instructions are installed or not; so why not allow
>> disabling apie. I suppose from the perspective of consistency, since the
>> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
>> both attributes, then it probably makes sense to add that check to
>> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
>> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
>> regardless of whether AP instructions are available. It boils down to
>> whether APIE needs to be dynamically disabled at some point when it has
>> been enabled. The only case I can think of where that may be necessary
>> is if a guest is migrated to a system without AP instructions. I don't
>> think that can happen and may even be protected against precisely
>> because the VM attributes won't be available on the target system due to
>> no AP instructions. What say you?
>>
> David,
>
> I'm sorry, I misinterpreted what you were asking for. Check out the
> fixup! patch below and let me know if that is what you are looking for.
> If so, I will integrate that change and post v11 tomorrow (Tuesday 9/24).
>
> -----------------------------------8<-----------------------------------
>
> From: Tony Krowiak <akrowiak@linux.ibm.com>
> Date: Mon, 24 Sep 2018 14:18:37 -0400
> Subject: [FIXUP v10] fixup! KVM: s390: device attrs to enable/disable AP
> interpretation
>
> ---
> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6654bb1fc26a..a528558baa78 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -901,6 +901,10 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm,
> struct kvm_device_attr *attr)
> kvm->arch.crypto.apie = 1;
> break;
> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> + if (!ap_instructions_available()) {
> + mutex_unlock(&kvm->lock);
> + return -EOPNOTSUPP;
> + }
> kvm->arch.crypto.apie = 0;
> break;
> default:
> @@ -1509,9 +1513,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
> struct kvm_device_attr *attr)
> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
> + ret = 0;
> + break;
> case KVM_S390_VM_CRYPTO_ENABLE_APIE:
> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> - ret = 0;
> + ret = ap_instructions_available();
Just a little remark, I guess we want to report 0 if available and
-ENXIO if not.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-25 7:32 ` David Hildenbrand
@ 2018-09-25 13:26 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-25 13:26 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/25/2018 03:32 AM, David Hildenbrand wrote:
> On 24/09/2018 20:42, Tony Krowiak wrote:
>> On 09/24/2018 12:25 PM, Tony Krowiak wrote:
>>> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>>
>> (...)
>>
>>>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>>>> only if supported by HW? (ap_instructions_available)
>>>
>>> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
>>> supported by HW, I assume you are talking about
>>> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
>>> ap_instructions_available() for disabling APIE because I didn't
>>> think it necessary given that ECA.28 will be set to 0 (intercept) by
>>> default, whether AP instructions are installed or not; so why not allow
>>> disabling apie. I suppose from the perspective of consistency, since the
>>> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
>>> both attributes, then it probably makes sense to add that check to
>>> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
>>> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
>>> regardless of whether AP instructions are available. It boils down to
>>> whether APIE needs to be dynamically disabled at some point when it has
>>> been enabled. The only case I can think of where that may be necessary
>>> is if a guest is migrated to a system without AP instructions. I don't
>>> think that can happen and may even be protected against precisely
>>> because the VM attributes won't be available on the target system due to
>>> no AP instructions. What say you?
>>>
>> David,
>>
>> I'm sorry, I misinterpreted what you were asking for. Check out the
>> fixup! patch below and let me know if that is what you are looking for.
>> If so, I will integrate that change and post v11 tomorrow (Tuesday 9/24).
>>
>> -----------------------------------8<-----------------------------------
>>
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>> Date: Mon, 24 Sep 2018 14:18:37 -0400
>> Subject: [FIXUP v10] fixup! KVM: s390: device attrs to enable/disable AP
>> interpretation
>>
>> ---
>> arch/s390/kvm/kvm-s390.c | 9 ++++++++-
>> 1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 6654bb1fc26a..a528558baa78 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -901,6 +901,10 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm,
>> struct kvm_device_attr *attr)
>> kvm->arch.crypto.apie = 1;
>> break;
>> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>> + if (!ap_instructions_available()) {
>> + mutex_unlock(&kvm->lock);
>> + return -EOPNOTSUPP;
>> + }
>> kvm->arch.crypto.apie = 0;
>> break;
>> default:
>> @@ -1509,9 +1513,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
>> struct kvm_device_attr *attr)
>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>> + ret = 0;
>> + break;
>> case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>> case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>> - ret = 0;
>> + ret = ap_instructions_available();
>
> Just a little remark, I guess we want to report 0 if available and
> -ENXIO if not.
That makes sense ... I'll fix it.
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 16:25 ` Tony Krowiak
2018-09-24 18:42 ` Tony Krowiak
@ 2018-09-24 18:46 ` David Hildenbrand
2018-09-25 13:31 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 18:46 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 24/09/2018 18:25, Tony Krowiak wrote:
> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>> On 22/09/2018 01:40, Tony Krowiak wrote:
>>> On 09/17/2018 04:51 AM, David Hildenbrand wrote:
>>>> Am 12.09.18 um 21:43 schrieb Tony Krowiak:
>>>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>>
>>>>> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
>>>>> to enable or disable AP instruction interpretation from userspace
>>>>> via the KVM_SET_DEVICE_ATTR ioctl:
>>>>>
>>>>> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
>>>>> interpretation of AP instructions executed on the guest.
>>>>>
>>>>> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
>>>>> interpretation of AP instructions executed on the guest. In this
>>>>> case the instructions will be intercepted and pass through to
>>>>> the guest.
>>>>>
>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>> ---
>>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>>> arch/s390/include/uapi/asm/kvm.h | 2 ++
>>>>> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
>>>>> 3 files changed, 26 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>>> index b32bd1b..36d3531 100644
>>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>>> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
>>>>> __u32 crycbd;
>>>>> __u8 aes_kw;
>>>>> __u8 dea_kw;
>>>>> + __u8 apie;
>>>>> };
>>>>>
>>>>> #define APCB0_MASK_SIZE 1
>>>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>>>> index 8c23afc..a8dbd90 100644
>>>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>>>> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
>>>>> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
>>>>> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
>>>>> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
>>>>> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
>>>>> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>>>>>
>>>>> /* kvm attributes for migration mode */
>>>>> #define KVM_S390_VM_MIGRATION_STOP 0
>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>>> index 2cdd980..286c2e0 100644
>>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>>> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>>>>
>>>>> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>> {
>>>>> - if (!test_kvm_facility(kvm, 76))
>>>>> - return -EINVAL;
>>>>> -
>>>>> mutex_lock(&kvm->lock);
>>>>> switch (attr->attr) {
>>>>> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>> + return -EINVAL;
>>>>> get_random_bytes(
>>>>> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
>>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>>> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
>>>>> break;
>>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>> + return -EINVAL;
>>>>> get_random_bytes(
>>>>> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
>>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>>> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
>>>>> break;
>>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>> + return -EINVAL;
>>>>> kvm->arch.crypto.aes_kw = 0;
>>>>> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
>>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
>>>>> break;
>>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>> + return -EINVAL;
>>>>> kvm->arch.crypto.dea_kw = 0;
>>>>> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
>>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
>>>>> break;
>>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>>> + if (!ap_instructions_available()) {
>>>>> + mutex_unlock(&kvm->lock);
>>>>> + return -EOPNOTSUPP;
>>>>> + }
>>>>> + kvm->arch.crypto.apie = 1;
>>>>> + break;
>>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>>> + kvm->arch.crypto.apie = 0;
>>>>> + break;
>>>>> default:
>>>>> mutex_unlock(&kvm->lock);
>>>>> return -ENXIO;
>>>>> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>>
>>>> As also replied to the QEMU series, could we indicate
>>>> KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
>>>> KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
>>>> so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
>>>>
>>>> KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
>>>> (never checked, we only care about apie).
>>>
>>> After much discussion with Halil and a few exchanges with you, we
>>> decided to go ahead and accept your suggestion to get rid of
>>> KVM_S390_VM_CPU_FEAT and keep the VM device attributes to enable/disable
>>> apie.
>>>
>>> To that end, I responded to patches 03/26, 11/26 and 25/26 with fixup!
>>> patches that show the KVM/kernel changes that will be necessary to get
>>> rid of KVM_S390_VM_CPU_FEAT and use apie to control ECA.28. I did that
>>> to generate discussion in v10 rather than waiting until v11 for
>>> comments. I make no guarantees that those fixup! patches will
>>> successfully apply should you have a v10 branch generated from this
>>> patch series you want to update.
>>>
>>
>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>> only if supported by HW? (ap_instructions_available)
>
> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
> supported by HW, I assume you are talking about
> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
> ap_instructions_available() for disabling APIE because I didn't
> think it necessary given that ECA.28 will be set to 0 (intercept) by
> default, whether AP instructions are installed or not; so why not allow
> disabling apie. I suppose from the perspective of consistency, since the
> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
> both attributes, then it probably makes sense to add that check to
> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
> regardless of whether AP instructions are available. It boils down to
> whether APIE needs to be dynamically disabled at some point when it has
> been enabled. The only case I can think of where that may be necessary
> is if a guest is migrated to a system without AP instructions. I don't
> think that can happen and may even be protected against precisely
> because the VM attributes won't be available on the target system due to
> no AP instructions. What say you?
>
>>
>
Just so we're on the same page, I am talking about exposing, I talk
about indicating the attribute:
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 03c23045527f..40924fe05bdf 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1491,6 +1491,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
struct kvm_device_attr *attr)
case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
ret = 0;
break;
+ case KVM_S390_VM_CRYPTO_ENABLE_APIE:
+ case KVM_S390_VM_CRYPTO_DISABLE_APIE:
+ ret = -ENXIO;
+ if (ap_instructions_available())
+ ret = 0;
default:
ret = -ENXIO;
break;
KVM_S390_VM_CRYPTO_DISABLE_APIE can either be handled like
KVM_S390_VM_CRYPTO_ENABLE_APIE (return -EOPNOTSUPP) when setting or
always be allowed. I'll leave that up to you. But as it is completely
useless without ap_instructions_available() /
KVM_S390_VM_CRYPTO_ENABLE_APIE , we might as well also just not expose
it then.
--
Thanks,
David / dhildenb
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation
2018-09-24 18:46 ` David Hildenbrand
@ 2018-09-25 13:31 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-25 13:31 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/24/2018 02:46 PM, David Hildenbrand wrote:
> On 24/09/2018 18:25, Tony Krowiak wrote:
>> On 09/24/2018 07:23 AM, David Hildenbrand wrote:
>>> On 22/09/2018 01:40, Tony Krowiak wrote:
>>>> On 09/17/2018 04:51 AM, David Hildenbrand wrote:
>>>>> Am 12.09.18 um 21:43 schrieb Tony Krowiak:
>>>>>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>>>
>>>>>> Introduces two new VM crypto device attributes (KVM_S390_VM_CRYPTO)
>>>>>> to enable or disable AP instruction interpretation from userspace
>>>>>> via the KVM_SET_DEVICE_ATTR ioctl:
>>>>>>
>>>>>> * The KVM_S390_VM_CRYPTO_ENABLE_APIE attribute enables hardware
>>>>>> interpretation of AP instructions executed on the guest.
>>>>>>
>>>>>> * The KVM_S390_VM_CRYPTO_DISABLE_APIE attribute disables hardware
>>>>>> interpretation of AP instructions executed on the guest. In this
>>>>>> case the instructions will be intercepted and pass through to
>>>>>> the guest.
>>>>>>
>>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>>>>>> ---
>>>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>>>> arch/s390/include/uapi/asm/kvm.h | 2 ++
>>>>>> arch/s390/kvm/kvm-s390.c | 27 +++++++++++++++++++++++----
>>>>>> 3 files changed, 26 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>>>> index b32bd1b..36d3531 100644
>>>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>>>> @@ -719,6 +719,7 @@ struct kvm_s390_crypto {
>>>>>> __u32 crycbd;
>>>>>> __u8 aes_kw;
>>>>>> __u8 dea_kw;
>>>>>> + __u8 apie;
>>>>>> };
>>>>>>
>>>>>> #define APCB0_MASK_SIZE 1
>>>>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>>>>> index 8c23afc..a8dbd90 100644
>>>>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>>>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>>>>> @@ -161,6 +161,8 @@ struct kvm_s390_vm_cpu_subfunc {
>>>>>> #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
>>>>>> #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
>>>>>> #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
>>>>>> +#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
>>>>>> +#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
>>>>>>
>>>>>> /* kvm attributes for migration mode */
>>>>>> #define KVM_S390_VM_MIGRATION_STOP 0
>>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>>>> index 2cdd980..286c2e0 100644
>>>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>>>> @@ -856,12 +856,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
>>>>>>
>>>>>> static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>>> {
>>>>>> - if (!test_kvm_facility(kvm, 76))
>>>>>> - return -EINVAL;
>>>>>> -
>>>>>> mutex_lock(&kvm->lock);
>>>>>> switch (attr->attr) {
>>>>>> case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
>>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>>> + return -EINVAL;
>>>>>> get_random_bytes(
>>>>>> kvm->arch.crypto.crycb->aes_wrapping_key_mask,
>>>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>>>> @@ -869,6 +868,8 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
>>>>>> break;
>>>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>>> + return -EINVAL;
>>>>>> get_random_bytes(
>>>>>> kvm->arch.crypto.crycb->dea_wrapping_key_mask,
>>>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>>>> @@ -876,17 +877,31 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>>> VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
>>>>>> break;
>>>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>>> + return -EINVAL;
>>>>>> kvm->arch.crypto.aes_kw = 0;
>>>>>> memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
>>>>>> sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
>>>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
>>>>>> break;
>>>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>>>> + if (!test_kvm_facility(kvm, 76))
>>>>>> + return -EINVAL;
>>>>>> kvm->arch.crypto.dea_kw = 0;
>>>>>> memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
>>>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
>>>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
>>>>>> break;
>>>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>>>> + if (!ap_instructions_available()) {
>>>>>> + mutex_unlock(&kvm->lock);
>>>>>> + return -EOPNOTSUPP;
>>>>>> + }
>>>>>> + kvm->arch.crypto.apie = 1;
>>>>>> + break;
>>>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>>>> + kvm->arch.crypto.apie = 0;
>>>>>> + break;
>>>>>> default:
>>>>>> mutex_unlock(&kvm->lock);
>>>>>> return -ENXIO;
>>>>>> @@ -1493,6 +1508,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>>>>>> case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
>>>>>> case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
>>>>>> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
>>>>>> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
>>>>>> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
>>>>>
>>>>> As also replied to the QEMU series, could we indicate
>>>>> KVM_S390_VM_CRYPTO_ENABLE_APIE (and maybe
>>>>> KVM_S390_VM_CRYPTO_DISABLE_APIE) only with ap_instructions_available(),
>>>>> so we can avoid the additional KVM_S390_VM_CPU_FEAT_AP?
>>>>>
>>>>> KVM_S390_VM_CPU_FEAT_AP is right now completely unused in KVM otherwise
>>>>> (never checked, we only care about apie).
>>>>
>>>> After much discussion with Halil and a few exchanges with you, we
>>>> decided to go ahead and accept your suggestion to get rid of
>>>> KVM_S390_VM_CPU_FEAT and keep the VM device attributes to enable/disable
>>>> apie.
>>>>
>>>> To that end, I responded to patches 03/26, 11/26 and 25/26 with fixup!
>>>> patches that show the KVM/kernel changes that will be necessary to get
>>>> rid of KVM_S390_VM_CPU_FEAT and use apie to control ECA.28. I did that
>>>> to generate discussion in v10 rather than waiting until v11 for
>>>> comments. I make no guarantees that those fixup! patches will
>>>> successfully apply should you have a v10 branch generated from this
>>>> patch series you want to update.
>>>>
>>>
>>> Will you also fixup this patch to expose KVM_S390_VM_CRYPTO_ENABLE_APIE
>>> only if supported by HW? (ap_instructions_available)
>>
>> Given that this patch DOES expose KVM_S390_VM_CRYPTO_ENABLE_APIE only if
>> supported by HW, I assume you are talking about
>> KVM_S390_VM_CRYPTO_DISABLE_APIE. I didn't check
>> ap_instructions_available() for disabling APIE because I didn't
>> think it necessary given that ECA.28 will be set to 0 (intercept) by
>> default, whether AP instructions are installed or not; so why not allow
>> disabling apie. I suppose from the perspective of consistency, since the
>> kvm_s390_vm_has_attr() function checks ap_instructions_available() for
>> both attributes, then it probably makes sense to add that check to
>> KVM_S390_VM_CRYPTO_DISABLE_APIE here. Then again, we could make a change
>> in ap_instructions_available() to allow KVM_S390_VM_CRYPTO_DISABLE_APIE
>> regardless of whether AP instructions are available. It boils down to
>> whether APIE needs to be dynamically disabled at some point when it has
>> been enabled. The only case I can think of where that may be necessary
>> is if a guest is migrated to a system without AP instructions. I don't
>> think that can happen and may even be protected against precisely
>> because the VM attributes won't be available on the target system due to
>> no AP instructions. What say you?
>>
>>>
>>
>
> Just so we're on the same page, I am talking about exposing, I talk
> about indicating the attribute:
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 03c23045527f..40924fe05bdf 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1491,6 +1491,11 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm,
> struct kvm_device_attr *attr)
> case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
> ret = 0;
> break;
> + case KVM_S390_VM_CRYPTO_ENABLE_APIE:
> + case KVM_S390_VM_CRYPTO_DISABLE_APIE:
> + ret = -ENXIO;
> + if (ap_instructions_available())
> + ret = 0;
> default:
> ret = -ENXIO;
> break;
>
> KVM_S390_VM_CRYPTO_DISABLE_APIE can either be handled like
> KVM_S390_VM_CRYPTO_ENABLE_APIE (return -EOPNOTSUPP) when setting or
> always be allowed. I'll leave that up to you. But as it is completely
> useless without ap_instructions_available() /
> KVM_S390_VM_CRYPTO_ENABLE_APIE , we might as well also just not expose
> it then.
We are on the same page.
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (23 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 24/26] KVM: s390: device attrs to enable/disable AP interpretation Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-21 23:31 ` Tony Krowiak
2018-09-12 19:43 ` [PATCH v10 26/26] s390: doc: detailed specifications " Tony Krowiak
` (2 subsequent siblings)
27 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
Introduces a new CPU model feature and two CPU model
facilities to support AP virtualization for KVM guests.
CPU model feature:
The KVM_S390_VM_CPU_FEAT_AP feature indicates that
AP instructions are available on the guest. This
feature will be enabled by the kernel only if the AP
instructions are installed on the linux host. This feature
must be specifically turned on for the KVM guest from
userspace to use the VFIO AP device driver for guest
access to AP devices.
CPU model facilities:
1. AP Query Configuration Information (QCI) facility is installed.
This is indicated by setting facilities bit 12 for
the guest. The kernel will not enable this facility
for the guest if it is not set on the host.
If this facility is not set for the KVM guest, then only
APQNs with an APQI less than 16 will be used by a Linux
guest regardless of the matrix configuration for the virtual
machine. This is a limitation of the Linux AP bus.
2. AP Facilities Test facility (APFT) is installed.
This is indicated by setting facilities bit 15 for
the guest. The kernel will not enable this facility for
the guest if it is not set on the host.
If this facility is not set for the KVM guest, then no
AP devices will be available to the guest regardless of
the guest's matrix configuration for the virtual
machine. This is a limitation of the Linux AP bus.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Tested-by: Michael Mueller <mimu@linux.ibm.com>
Tested-by: Farhan Ali <alifm@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
arch/s390/kvm/kvm-s390.c | 5 +++++
arch/s390/tools/gen_facilities.c | 2 ++
2 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 286c2e0..f0b8e2a 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -371,6 +371,11 @@ static void kvm_s390_cpu_feat_init(void)
if (MACHINE_HAS_ESOP)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
+
+ /* Check if AP instructions installed on host */
+ if (ap_instructions_available())
+ allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
+
/*
* We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
* 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
index 0c85aed..fd788e0 100644
--- a/arch/s390/tools/gen_facilities.c
+++ b/arch/s390/tools/gen_facilities.c
@@ -106,6 +106,8 @@ struct facility_def {
.name = "FACILITIES_KVM_CPUMODEL",
.bits = (int[]){
+ 12, /* AP Query Configuration Information */
+ 15, /* AP Facilities Test */
156, /* etoken facility */
-1 /* END */
}
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization
2018-09-12 19:43 ` [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization Tony Krowiak
@ 2018-09-21 23:31 ` Tony Krowiak
2018-09-24 8:33 ` David Hildenbrand
0 siblings, 1 reply; 87+ messages in thread
From: Tony Krowiak @ 2018-09-21 23:31 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 09/12/2018 03:43 PM, Tony Krowiak wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Introduces a new CPU model feature and two CPU model
> facilities to support AP virtualization for KVM guests.
>
> CPU model feature:
>
> The KVM_S390_VM_CPU_FEAT_AP feature indicates that
> AP instructions are available on the guest. This
> feature will be enabled by the kernel only if the AP
> instructions are installed on the linux host. This feature
> must be specifically turned on for the KVM guest from
> userspace to use the VFIO AP device driver for guest
> access to AP devices.
>
> CPU model facilities:
>
> 1. AP Query Configuration Information (QCI) facility is installed.
>
> This is indicated by setting facilities bit 12 for
> the guest. The kernel will not enable this facility
> for the guest if it is not set on the host.
>
> If this facility is not set for the KVM guest, then only
> APQNs with an APQI less than 16 will be used by a Linux
> guest regardless of the matrix configuration for the virtual
> machine. This is a limitation of the Linux AP bus.
>
> 2. AP Facilities Test facility (APFT) is installed.
>
> This is indicated by setting facilities bit 15 for
> the guest. The kernel will not enable this facility for
> the guest if it is not set on the host.
>
> If this facility is not set for the KVM guest, then no
> AP devices will be available to the guest regardless of
> the guest's matrix configuration for the virtual
> machine. This is a limitation of the Linux AP bus.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
> Reviewed-by: David Hildenbrand <david@redhat.com>
> Tested-by: Michael Mueller <mimu@linux.ibm.com>
> Tested-by: Farhan Ali <alifm@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> arch/s390/kvm/kvm-s390.c | 5 +++++
> arch/s390/tools/gen_facilities.c | 2 ++
> 2 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 286c2e0..f0b8e2a 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -371,6 +371,11 @@ static void kvm_s390_cpu_feat_init(void)
>
> if (MACHINE_HAS_ESOP)
> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
> +
> + /* Check if AP instructions installed on host */
> + if (ap_instructions_available())
> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
> +
> /*
> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
> index 0c85aed..fd788e0 100644
> --- a/arch/s390/tools/gen_facilities.c
> +++ b/arch/s390/tools/gen_facilities.c
> @@ -106,6 +106,8 @@ struct facility_def {
>
> .name = "FACILITIES_KVM_CPUMODEL",
> .bits = (int[]){
> + 12, /* AP Query Configuration Information */
> + 15, /* AP Facilities Test */
> 156, /* etoken facility */
> -1 /* END */
> }
>
The fixup! patch below modifies this patch (25/26) to illustrate how
David's recommendation will be implemented for v11 of the series. It
is one of three fixup! patches (the other two are in responses to
03/26 and 11/26) included to generate discussion in v10 rather than
waiting until v11 for comments.
-----------------------------------8<-----------------------------------
From: Tony Krowiak <akrowiak@linux.ibm.com>
Date: Thu, 20 Sep 2018 13:28:07 -0400
Subject: [FIXUP v10] fixup!: KVM: s390: CPU model support for AP
virtualization
---
arch/s390/kvm/kvm-s390.c | 4 ----
1 files changed, 0 insertions(+), 4 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index a3a7cd9..ff38251 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -372,10 +372,6 @@ static void kvm_s390_cpu_feat_init(void)
if (MACHINE_HAS_ESOP)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
- /* Check if AP instructions installed on host */
- if (ap_instructions_available())
- allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
-
/*
* We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
* 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization
2018-09-21 23:31 ` Tony Krowiak
@ 2018-09-24 8:33 ` David Hildenbrand
0 siblings, 0 replies; 87+ messages in thread
From: David Hildenbrand @ 2018-09-24 8:33 UTC (permalink / raw)
To: Tony Krowiak, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
frankja
On 22/09/2018 01:31, Tony Krowiak wrote:
> On 09/12/2018 03:43 PM, Tony Krowiak wrote:
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Introduces a new CPU model feature and two CPU model
>> facilities to support AP virtualization for KVM guests.
>>
>> CPU model feature:
>>
>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that
>> AP instructions are available on the guest. This
>> feature will be enabled by the kernel only if the AP
>> instructions are installed on the linux host. This feature
>> must be specifically turned on for the KVM guest from
>> userspace to use the VFIO AP device driver for guest
>> access to AP devices.
>>
>> CPU model facilities:
>>
>> 1. AP Query Configuration Information (QCI) facility is installed.
>>
>> This is indicated by setting facilities bit 12 for
>> the guest. The kernel will not enable this facility
>> for the guest if it is not set on the host.
>>
>> If this facility is not set for the KVM guest, then only
>> APQNs with an APQI less than 16 will be used by a Linux
>> guest regardless of the matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> 2. AP Facilities Test facility (APFT) is installed.
>>
>> This is indicated by setting facilities bit 15 for
>> the guest. The kernel will not enable this facility for
>> the guest if it is not set on the host.
>>
>> If this facility is not set for the KVM guest, then no
>> AP devices will be available to the guest regardless of
>> the guest's matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
>> Reviewed-by: David Hildenbrand <david@redhat.com>
>> Tested-by: Michael Mueller <mimu@linux.ibm.com>
>> Tested-by: Farhan Ali <alifm@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>> arch/s390/kvm/kvm-s390.c | 5 +++++
>> arch/s390/tools/gen_facilities.c | 2 ++
>> 2 files changed, 7 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 286c2e0..f0b8e2a 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -371,6 +371,11 @@ static void kvm_s390_cpu_feat_init(void)
>>
>> if (MACHINE_HAS_ESOP)
>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>> +
>> + /* Check if AP instructions installed on host */
>> + if (ap_instructions_available())
>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>> +
>> /*
>> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
>> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>> index 0c85aed..fd788e0 100644
>> --- a/arch/s390/tools/gen_facilities.c
>> +++ b/arch/s390/tools/gen_facilities.c
>> @@ -106,6 +106,8 @@ struct facility_def {
>>
>> .name = "FACILITIES_KVM_CPUMODEL",
>> .bits = (int[]){
>> + 12, /* AP Query Configuration Information */
>> + 15, /* AP Facilities Test */
>> 156, /* etoken facility */
>> -1 /* END */
>> }
>>
>
> The fixup! patch below modifies this patch (25/26) to illustrate how
> David's recommendation will be implemented for v11 of the series. It
> is one of three fixup! patches (the other two are in responses to
> 03/26 and 11/26) included to generate discussion in v10 rather than
> waiting until v11 for comments.
>
> -----------------------------------8<-----------------------------------
>
> From: Tony Krowiak <akrowiak@linux.ibm.com>
> Date: Thu, 20 Sep 2018 13:28:07 -0400
> Subject: [FIXUP v10] fixup!: KVM: s390: CPU model support for AP
> virtualization
>
> ---
> arch/s390/kvm/kvm-s390.c | 4 ----
> 1 files changed, 0 insertions(+), 4 deletions(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index a3a7cd9..ff38251 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -372,10 +372,6 @@ static void kvm_s390_cpu_feat_init(void)
> if (MACHINE_HAS_ESOP)
> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>
> - /* Check if AP instructions installed on host */
> - if (ap_instructions_available())
> - allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
> -
> /*
> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
>
Yes, looks sane.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v10 26/26] s390: doc: detailed specifications for AP virtualization
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (24 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 25/26] KVM: s390: CPU model support for AP virtualization Tony Krowiak
@ 2018-09-12 19:43 ` Tony Krowiak
2018-09-24 10:10 ` [PATCH v10 00/26] guest dedicated crypto adapters Christian Borntraeger
2018-09-24 11:49 ` Cornelia Huck
27 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-12 19:43 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, berrange, fiuczy, buendgen,
akrowiak, frankja, Tony Krowiak
From: Tony Krowiak <akrowiak@linux.ibm.com>
This patch provides documentation describing the AP architecture and
design concepts behind the virtualization of AP devices. It also
includes an example of how to configure AP devices for exclusive
use of KVM guests.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
Documentation/s390/vfio-ap.txt | 778 ++++++++++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
2 files changed, 779 insertions(+), 0 deletions(-)
create mode 100644 Documentation/s390/vfio-ap.txt
diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
new file mode 100644
index 0000000..7bdf649
--- /dev/null
+++ b/Documentation/s390/vfio-ap.txt
@@ -0,0 +1,778 @@
+Introduction:
+============
+The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
+of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
+The AP devices provide cryptographic functions to all CPUs assigned to a
+linux system running in an IBM Z system LPAR.
+
+The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
+is to make AP cards available to KVM guests using the VFIO mediated device
+framework. This implementation relies considerably on the s390 virtualization
+facilities which do most of the hard work of providing direct access to AP
+devices.
+
+AP Architectural Overview:
+=========================
+To facilitate the comprehension of the design, let's start with some
+definitions:
+
+* AP adapter
+
+ An AP adapter is an IBM Z adapter card that can perform cryptographic
+ functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
+ assigned to the LPAR in which a linux host is running will be available to
+ the linux host. Each adapter is identified by a number from 0 to 255. When
+ installed, an AP adapter is accessed by AP instructions executed by any CPU.
+
+ The AP adapter cards are assigned to a given LPAR via the system's Activation
+ Profile which can be edited via the HMC. When the linux host system is IPL'd
+ in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
+ creates a sysfs device for each assigned adapter. For example, if AP adapters
+ 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
+ sysfs device entries:
+
+ /sys/devices/ap/card04
+ /sys/devices/ap/card0a
+
+ Symbolic links to these devices will also be created in the AP bus devices
+ sub-directory:
+
+ /sys/bus/ap/devices/[card04]
+ /sys/bus/ap/devices/[card04]
+
+* AP domain
+
+ An adapter is partitioned into domains. An adapter can hold up to 256 domains
+ depending upon the adapter type and hardware configuration. A domain is
+ identified by a number from 0 to 255. A domain can be thought of as a set of
+ hardware registers and memory used for processing AP commands. A domain can be
+ configured with a secure private key used for clear key encryption. A domain
+ is classified in one of two ways depending upon how it may be accessed:
+
+ * Usage domains are domains that are targeted by an AP instruction to
+ process an AP command.
+
+ * Control domains are domains that are changed by an AP command sent to a
+ usage domain; for example, to set the secure private key for the control
+ domain.
+
+ The AP usage and control domains are assigned to a given LPAR via the system's
+ Activation Profile which can be edited via the HMC. When a linux host system
+ is IPL'd in the LPAR, the AP bus module detects the AP usage and control
+ domains assigned to the LPAR. The domain number of each usage domain and
+ adapter number of each AP adapter are combined to create AP queue devices
+ (see AP Queue section below). The domain number of each control domain will be
+ represented in a bitmask and stored in a sysfs file
+ /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
+ significant bit, correspond to domains 0-255.
+
+* AP Queue
+
+ An AP queue is the means by which an AP command is sent to a usage domain
+ inside a specific adapter. An AP queue is identified by a tuple
+ comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
+ APQI corresponds to a given usage domain number within the adapter. This tuple
+ forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
+ instructions include a field containing the APQN to identify the AP queue to
+ which the AP command is to be sent for processing.
+
+ The AP bus will create a sysfs device for each APQN that can be derived from
+ the cross product of the AP adapter and usage domain numbers detected when the
+ AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
+ domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
+ following sysfs entries:
+
+ /sys/devices/ap/card04/04.0006
+ /sys/devices/ap/card04/04.0047
+ /sys/devices/ap/card0a/0a.0006
+ /sys/devices/ap/card0a/0a.0047
+
+ The following symbolic links to these devices will be created in the AP bus
+ devices subdirectory:
+
+ /sys/bus/ap/devices/[04.0006]
+ /sys/bus/ap/devices/[04.0047]
+ /sys/bus/ap/devices/[0a.0006]
+ /sys/bus/ap/devices/[0a.0047]
+
+* AP Instructions:
+
+ There are three AP instructions:
+
+ * NQAP: to enqueue an AP command-request message to a queue
+ * DQAP: to dequeue an AP command-reply message from a queue
+ * PQAP: to administer the queues
+
+ AP instructions identify the domain that is targeted to process the AP
+ command; this must be one of the usage domains. An AP command may modify a
+ domain that is not one of the usage domains, but the modified domain
+ must be one of the control domains.
+
+AP and SIE:
+==========
+Let's now take a look at how AP instructions executed on a guest are interpreted
+by the hardware.
+
+A satellite control block called the Crypto Control Block (CRYCB) is attached to
+our main hardware virtualization control block. The CRYCB contains three fields
+to identify the adapters, usage domains and control domains assigned to the KVM
+guest:
+
+* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
+ to the KVM guest. Each bit in the mask, from most significant to least
+ significant bit, corresponds to an APID from 0-255. If a bit is set, the
+ corresponding adapter is valid for use by the KVM guest.
+
+* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
+ assigned to the KVM guest. Each bit in the mask, from most significant to
+ least significant bit, corresponds to an AP queue index (APQI) from 0-255. If
+ a bit is set, the corresponding queue is valid for use by the KVM guest.
+
+* The AP Domain Mask field is a bit mask that identifies the AP control domains
+ assigned to the KVM guest. The ADM bit mask controls which domains can be
+ changed by an AP command-request message sent to a usage domain from the
+ guest. Each bit in the mask, from least significant to most significant bit,
+ corresponds to a domain from 0-255. If a bit is set, the corresponding domain
+ can be modified by an AP command-request message sent to a usage domain.
+
+If you recall from the description of an AP Queue, AP instructions include
+an APQN to identify the AP queue to which an AP command-request message is to be
+sent (NQAP and PQAP instructions), or from which a command-reply message is to
+be received (DQAP instruction). The validity of an APQN is defined by the matrix
+calculated from the APM and AQM; it is the cross product of all assigned adapter
+numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1
+and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6),
+(2,5) and (2,6) will be valid for the guest.
+
+The APQNs can provide secure key functionality - i.e., a private key is stored
+on the adapter card for each of its domains - so each APQN must be assigned to
+at most one guest or to the linux host.
+
+ Example 1: Valid configuration:
+ ------------------------------
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1,2 domain 7
+
+ This is valid because both guests have a unique set of APQNs:
+ Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
+ Guest2 has APQNs (1,7), (2,7)
+
+ Example 2: Valid configuration:
+ ------------------------------
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapters 3,4 domains 5,6
+
+ This is also valid because both guests have a unique set of APQNs:
+ Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
+ Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
+
+ Example 3: Invalid configuration:
+ --------------------------------
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1 domains 6,7
+
+ This is an invalid configuration because both guests have access to
+ APQN (1,6).
+
+The Design:
+===========
+The design introduces three new objects:
+
+1. AP matrix device
+2. VFIO AP device driver (vfio_ap.ko)
+3. VFIO AP mediated matrix pass-through device
+
+The VFIO AP device driver
+-------------------------
+The VFIO AP (vfio_ap) device driver serves the following purposes:
+
+1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
+
+2. Sets up the VFIO mediated device interfaces to manage a mediated matrix
+ device and creates the sysfs interfaces for assigning adapters, usage
+ domains, and control domains comprising the matrix for a KVM guest.
+
+3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
+ SIE state description to grant the guest access to a matrix of AP devices
+
+Reserve APQNs for exclusive use of KVM guests
+---------------------------------------------
+The following block diagram illustrates the mechanism by which APQNs are
+reserved:
+
+ +------------------+
+ 7 remove | |
+ +--------------------> cex4queue driver |
+ | | |
+ | +------------------+
+ |
+ |
+ | +------------------+ +-----------------+
+ | 5 register driver | | 3 create | |
+ | +----------------> Device core +----------> matrix device |
+ | | | | | |
+ | | +--------^---------+ +-----------------+
+ | | |
+ | | +-------------------+
+ | | +-----------------------------------+ |
+ | | | 4 register AP driver | | 2 register device
+ | | | | |
++--------+---+-v---+ +--------+-------+-+
+| | | |
+| ap_bus +--------------------- > vfio_ap driver |
+| | 8 probe | |
++--------^---------+ +--^--^------------+
+6 edit | | |
+ apmask | +-----------------------------+ | 9 mdev create
+ aqmask | | 1 modprobe |
++--------+-----+---+ +----------------+-+ +------------------+
+| | | |8 create | mediated |
+| admin | | VFIO device core |---------> matrix |
+| + | | | device |
++------+-+---------+ +--------^---------+ +--------^---------+
+ | | | |
+ | | 9 create vfio_ap-passthrough | |
+ | +------------------------------+ |
+ +-------------------------------------------------------------+
+ 10 assign adapter/domain/control domain
+
+The process for reserving an AP queue for use by a KVM guest is:
+
+1. The administrator loads the vfio_ap device driver
+2. The vfio-ap driver during its initialization will register a single 'matrix'
+ device with the device core. This will serve as the parent device for
+ all mediated matrix devices used to configure an AP matrix for a guest.
+3. The /sys/devices/vfio_ap/matrix device is created by the device core
+4 The vfio_ap device driver will register with the AP bus for AP queue devices
+ of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
+ driver's probe and remove callback interfaces. Devices older than CEX4 queues
+ are not supported to simplify the implementation by not needlessly
+ complicating the design by supporting older devices that will go out of
+ service in the relatively near future, and for which there are few older
+ systems around on which to test.
+5. The AP bus registers the vfio_ap device driver with the device core
+6. The administrator edits the AP adapter and queue masks to reserve AP queues
+ for use by the vfio_ap device driver.
+7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
+ default zcrypt cex4queue driver.
+8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
+ it.
+9. The administrator creates a passthrough type mediated matrix device to be
+ used by a guest
+10 The administrator assigns the adapters, usage domains and control domains
+ to be exclusively used by a guest.
+
+Set up the VFIO mediated device interfaces
+------------------------------------------
+The VFIO AP device driver utilizes the common interface of the VFIO mediated
+device core driver to:
+* Register an AP mediated bus driver to add a mediated matrix device to and
+ remove it from a VFIO group.
+* Create and destroy a mediated matrix device
+* Add a mediated matrix device to and remove it from the AP mediated bus driver
+* Add a mediated matrix device to and remove it from an IOMMU group
+
+The following high-level block diagram shows the main components and interfaces
+of the VFIO AP mediated matrix device driver:
+
+ +-------------+
+ | |
+ | +---------+ | mdev_register_driver() +--------------+
+ | | Mdev | +<-----------------------+ |
+ | | bus | | | vfio_mdev.ko |
+ | | driver | +----------------------->+ |<-> VFIO user
+ | +---------+ | probe()/remove() +--------------+ APIs
+ | |
+ | MDEV CORE |
+ | MODULE |
+ | mdev.ko |
+ | +---------+ | mdev_register_device() +--------------+
+ | |Physical | +<-----------------------+ |
+ | | device | | | vfio_ap.ko |<-> matrix
+ | |interface| +----------------------->+ | device
+ | +---------+ | callback +--------------+
+ +-------------+
+
+During initialization of the vfio_ap module, the matrix device is registered
+with an 'mdev_parent_ops' structure that provides the sysfs attribute
+structures, mdev functions and callback interfaces for managing the mediated
+matrix device.
+
+* sysfs attribute structures:
+ * supported_type_groups
+ The VFIO mediated device framework supports creation of user-defined
+ mediated device types. These mediated device types are specified
+ via the 'supported_type_groups' structure when a device is registered
+ with the mediated device framework. The registration process creates the
+ sysfs structures for each mediated device type specified in the
+ 'mdev_supported_types' sub-directory of the device being registered. Along
+ with the device type, the sysfs attributes of the mediated device type are
+ provided.
+
+ The VFIO AP device driver will register one mediated device type for
+ passthrough devices:
+ /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough
+ Only the read-only attributes required by the VFIO mdev framework will
+ be provided:
+ ... name
+ ... device_api
+ ... available_instances
+ ... device_api
+ Where:
+ * name: specifies the name of the mediated device type
+ * device_api: the mediated device type's API
+ * available_instances: the number of mediated matrix passthrough devices
+ that can be created
+ * device_api: specifies the VFIO API
+ * mdev_attr_groups
+ This attribute group identifies the user-defined sysfs attributes of the
+ mediated device. When a device is registered with the VFIO mediated device
+ framework, the sysfs attribute files identified in the 'mdev_attr_groups'
+ structure will be created in the mediated matrix device's directory. The
+ sysfs attributes for a mediated matrix device are:
+ * assign_adapter:
+ * unassign_adapter:
+ Write-only attributes for assigning/unassigning an AP adapter to/from the
+ mediated matrix device. To assign/unassign an adapter, the APID of the
+ adapter is echoed to the respective attribute file.
+ * assign_domain:
+ * unassign_domain:
+ Write-only attributes for assigning/unassigning an AP usage domain to/from
+ the mediated matrix device. To assign/unassign a domain, the domain
+ number of the the usage domain is echoed to the respective attribute
+ file.
+ * matrix:
+ A read-only file for displaying the APQNs derived from the cross product
+ of the adapter and domain numbers assigned to the mediated matrix device.
+ * assign_control_domain:
+ * unassign_control_domain:
+ Write-only attributes for assigning/unassigning an AP control domain
+ to/from the mediated matrix device. To assign/unassign a control domain,
+ the ID of the domain to be assigned/unassigned is echoed to the respective
+ attribute file.
+ * control_domains:
+ A read-only file for displaying the control domain numbers assigned to the
+ mediated matrix device.
+
+* functions:
+ * create:
+ allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
+ * Store the reference to the KVM structure for the guest using the mdev
+ * Store the AP matrix configuration for the adapters, domains, and control
+ domains assigned via the corresponding sysfs attributes files
+ * remove:
+ deallocates the mediated matrix device's ap_matrix_mdev structure. This will
+ be allowed only if a running guest is not using the mdev.
+
+* callback interfaces
+ * open:
+ The vfio_ap driver uses this callback to register a
+ VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
+ device. The open is invoked when QEMU connects the VFIO iommu group
+ for the mdev matrix device to the MDEV bus. Access to the KVM structure used
+ to configure the KVM guest is provided via this callback. The KVM structure,
+ is used to configure the guest's access to the AP matrix defined via the
+ mediated matrix device's sysfs attribute files.
+ * release:
+ unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
+ mdev matrix device and deconfigures the guest's AP matrix.
+
+Configure the APM, AQM and ADM in the CRYCB:
+-------------------------------------------
+Configuring the AP matrix for a KVM guest will be performed when the
+VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
+function is called when QEMU connects to KVM. The guest's AP matrix is
+configured via it's CRYCB by:
+* Setting the bits in the APM corresponding to the APIDs assigned to the
+ mediated matrix device via its 'assign_adapter' interface.
+* Setting the bits in the AQM corresponding to the domains assigned to the
+ mediated matrix device via its 'assign_domain' interface.
+* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
+ mediated matrix device via its 'assign_control_domains' interface.
+
+The CPU model features for AP
+-----------------------------
+The AP stack relies on the presence of the AP instructions as well as two
+facilities: The AP Facilities Test (APFT) facility; and the AP Query
+Configuration Information (QCI) facility. These features/facilities are made
+available to a KVM guest via the following CPU model features:
+
+1. ap: Indicates whether the AP instructions are installed on the guest. This
+ feature will be enabled by KVM only if the AP instructions are installed
+ on the host.
+
+2. apft: Indicates the APFT facility is available on the guest. This facility
+ can be made available to the guest only if it is available on the host (i.e.,
+ facility bit 15 is set).
+
+3. apqci: Indicates the AP QCI facility is available on the guest. This facility
+ can be made available to the guest only if it is available on the host (i.e.,
+ facility bit 12 is set).
+
+Note: If the user chooses to specify a CPU model different than the 'host'
+model to QEMU, the CPU model features and facilities need to be turned on
+explicitly; for example:
+
+ /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on
+
+A guest can be precluded from using AP features/facilities by turning them off
+explicitly; for example:
+
+ /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
+
+Note: If the APFT facility is turned off (apft=off) for the guest, the guest
+will not see any AP devices. The zcrypt device drivers that register for type 10
+and newer AP devices - i.e., the cex4card and cex4queue device drivers - need
+the APFT facility to ascertain the facilities installed on a given AP device. If
+the APFT facility is not installed on the guest, then the probe of device
+drivers will fail since only type 10 and newer devices can be configured for
+guest use.
+
+Example:
+=======
+Let's now provide an example to illustrate how KVM guests may be given
+access to AP facilities. For this example, we will show how to configure
+three guests such that executing the lszcrypt command on the guests would
+look like this:
+
+Guest1
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5C CCA-Coproc
+05.0004 CEX5C CCA-Coproc
+05.00ab CEX5C CCA-Coproc
+06 CEX5A Accelerator
+06.0004 CEX5A Accelerator
+06.00ab CEX5C CCA-Coproc
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5A Accelerator
+05.0047 CEX5A Accelerator
+05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171),
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+06 CEX5A Accelerator
+06.0047 CEX5A Accelerator
+06.00ff CEX5A Accelerator
+
+These are the steps:
+
+1. Install the vfio_ap module on the linux host. The dependency chain for the
+ vfio_ap module is:
+ * iommu
+ * s390
+ * zcrypt
+ * vfio
+ * vfio_mdev
+ * vfio_mdev_device
+ * KVM
+
+ To build the vfio_ap module, the kernel build must be configured with the
+ following Kconfig elements selected:
+ * IOMMU_SUPPORT
+ * S390
+ * ZCRYPT
+ * S390_AP_IOMMU
+ * VFIO
+ * VFIO_MDEV
+ * VFIO_MDEV_DEVICE
+ * KVM
+
+ If using make menuconfig select the following to build the vfio_ap module:
+ -> Device Drivers
+ -> IOMMU Hardware Support
+ select S390 AP IOMMU Support
+ -> VFIO Non-Privileged userspace driver framework
+ -> Mediated device driver frramework
+ -> VFIO driver for Mediated devices
+ -> I/O subsystem
+ -> VFIO support for AP devices
+
+2. Secure the AP queues to be used by the three guests so that the host can not
+ access them. To secure them, there are two sysfs files that specify
+ bitmasks marking a subset of the APQN range as 'usable by the default AP
+ queue device drivers' or 'not usable by the default device drivers' and thus
+ available for use by the vfio_ap device driver'. The sysfs files containing
+ the sysfs locations of the masks are:
+
+ /sys/bus/ap/apmask
+ /sys/bus/ap/aqmask
+
+ The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
+ (APID). Each bit in the mask, from most significant to least significant bit,
+ corresponds to an APID from 0-255. If a bit is set, the APID is marked as
+ usable only by the default AP queue device drivers; otherwise, the APID is
+ usable by the vfio_ap device driver.
+
+ The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
+ (APQI). Each bit in the mask, from most significant to least significant bit,
+ corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
+ usable only by the default AP queue device drivers; otherwise, the APQI is
+ usable by the vfio_ap device driver.
+
+ The APQN of each AP queue device assigned to the linux host is checked by the
+ AP bus against the set of APQNs derived from the cross product of APIDs
+ and APQIs marked as usable only by the default AP queue device drivers. If a
+ match is detected, only the default AP queue device drivers will be probed;
+ otherwise, the vfio_ap device driver will be probed.
+
+ By default, the two masks are set to reserve all APQNs for use by the default
+ AP queue device drivers. There are two ways the default masks can be changed:
+
+ 1. The masks can be changed at boot time with the kernel command line
+ like this:
+
+ ap.apmask=0xffff ap.aqmask=0x40
+
+ This would give these two pools:
+
+ default drivers pool: adapter 0-15, domain 1
+ alternate drivers pool: adapter 16-255, domains 2-255
+
+ 2. The sysfs mask files can also be edited by echoing a string into the
+ respective file in one of two formats:
+
+ * An absolute hex string starting with 0x - like "0x12345678" - sets
+ the mask. If the given string is shorter than the mask, it is padded
+ with 0s on the right. If the string is longer than the mask, the
+ operation is terminated with an error (EINVAL).
+
+ * A plus ('+') or minus ('-') followed by a numerical value. Valid
+ examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only
+ the corresponding bit in the mask is switched on ('+') or off ('-'). The
+ values may also be specified in a comma-separated list to switch more
+ than one bit on or off.
+
+ To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
+ 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
+ APQNs must be removed from the masks as follows:
+
+ echo -5,-6 > /sys/bus/ap/apmask
+
+ echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
+
+ This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
+ 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
+ sysfs directory for the vfio_ap device driver will now contain symbolic links
+ to the AP queue devices bound to it:
+
+ /sys/bus/ap
+ ... [drivers]
+ ...... [vfio_ap]
+ ......... [05.0004]
+ ......... [05.0047]
+ ......... [05.00ab]
+ ......... [05.00ff]
+ ......... [06.0004]
+ ......... [06.0047]
+ ......... [06.00ab]
+ ......... [06.00ff]
+
+ Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
+ can be bound to the vfio_ap device driver. The reason for this is to
+ simplify the implementation by not needlessly complicating the design by
+ supporting older devices that will go out of service in the relatively near
+ future and for which there are few older systems on which to test.
+
+ The administrator, therefore, must take care to secure only AP queues that
+ can be bound to the vfio_ap device driver. The device type for a given AP
+ queue device can be read from the parent card's sysfs directory. For example,
+ to see the hardware type of the queue 05.0004:
+
+ cat /sys/bus/ap/devices/card05/hwtype
+
+ The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
+ vfio_ap device driver.
+
+3. Create the mediated devices needed to configure the AP matrixes for the
+ three guests and to provide an interface to the vfio_ap driver for
+ use by the guests:
+
+ /sys/devices/vfio_ap/matrix/
+ --- [mdev_supported_types]
+ ------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
+ --------- create
+ --------- [devices]
+
+ To create the mediated devices for the three guests:
+
+ uuidgen > create
+ uuidgen > create
+ uuidgen > create
+
+ or
+
+ echo $uuid1 > create
+ echo $uuid2 > create
+ echo $uuid3 > create
+
+ This will create three mediated devices in the [devices] subdirectory named
+ after the UUID written to the create attribute file. We call them $uuid1,
+ $uuid2 and $uuid3 and this is the sysfs directory structure after creation:
+
+ /sys/devices/vfio_ap/matrix/
+ --- [mdev_supported_types]
+ ------ [vfio_ap-passthrough]
+ --------- [devices]
+ ------------ [$uuid1]
+ --------------- assign_adapter
+ --------------- assign_control_domain
+ --------------- assign_domain
+ --------------- matrix
+ --------------- unassign_adapter
+ --------------- unassign_control_domain
+ --------------- unassign_domain
+
+ ------------ [$uuid2]
+ --------------- assign_adapter
+ --------------- assign_control_domain
+ --------------- assign_domain
+ --------------- matrix
+ --------------- unassign_adapter
+ ----------------unassign_control_domain
+ ----------------unassign_domain
+
+ ------------ [$uuid3]
+ --------------- assign_adapter
+ --------------- assign_control_domain
+ --------------- assign_domain
+ --------------- matrix
+ --------------- unassign_adapter
+ ----------------unassign_control_domain
+ ----------------unassign_domain
+
+4. The administrator now needs to configure the matrixes for the mediated
+ devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
+
+ This is how the matrix is configured for Guest1:
+
+ echo 5 > assign_adapter
+ echo 6 > assign_adapter
+ echo 4 > assign_domain
+ echo 0xab > assign_domain
+
+ Control domains can similarly be assigned using the assign_control_domain
+ sysfs file.
+
+ If a mistake is made configuring an adapter, domain or control domain,
+ you can use the unassign_xxx files to unassign the adapter, domain or
+ control domain.
+
+ To display the matrix configuration for Guest1:
+
+ cat matrix
+
+ This is how the matrix is configured for Guest2:
+
+ echo 5 > assign_adapter
+ echo 0x47 > assign_domain
+ echo 0xff > assign_domain
+
+ This is how the matrix is configured for Guest3:
+
+ echo 6 > assign_adapter
+ echo 0x47 > assign_domain
+ echo 0xff > assign_domain
+
+ In order to successfully assign an adapter:
+
+ * The adapter number specified must represent a value from 0 up to the
+ maximum adapter number configured for the system. If an adapter number
+ higher than the maximum is specified, the operation will terminate with
+ an error (ENODEV).
+
+ * All APQNs that can be derived from the adapter ID and the IDs of
+ the previously assigned domains must be bound to the vfio_ap device
+ driver. If no domains have yet been assigned, then there must be at least
+ one APQN with the specified APID bound to the vfio_ap driver. If no such
+ APQNs are bound to the driver, the operation will terminate with an
+ error (EADDRNOTAVAIL).
+
+ No APQN that can be derived from the adapter ID and the IDs of the
+ previously assigned domains can be assigned to another mediated matrix
+ device. If an APQN is assigned to another mediated matrix device, the
+ operation will terminate with an error (EADDRINUSE).
+
+ In order to successfully assign a domain:
+
+ * The domain number specified must represent a value from 0 up to the
+ maximum domain number configured for the system. If a domain number
+ higher than the maximum is specified, the operation will terminate with
+ an error (ENODEV).
+
+ * All APQNs that can be derived from the domain ID and the IDs of
+ the previously assigned adapters must be bound to the vfio_ap device
+ driver. If no domains have yet been assigned, then there must be at least
+ one APQN with the specified APQI bound to the vfio_ap driver. If no such
+ APQNs are bound to the driver, the operation will terminate with an
+ error (EADDRNOTAVAIL).
+
+ No APQN that can be derived from the domain ID and the IDs of the
+ previously assigned adapters can be assigned to another mediated matrix
+ device. If an APQN is assigned to another mediated matrix device, the
+ operation will terminate with an error (EADDRINUSE).
+
+ In order to successfully assign a control domain, the domain number
+ specified must represent a value from 0 up to the maximum domain number
+ configured for the system. If a control domain number higher than the maximum
+ is specified, the operation will terminate with an error (ENODEV).
+
+5. Start Guest1:
+
+ /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
+
+7. Start Guest2:
+
+ /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
+
+7. Start Guest3:
+
+ /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
+
+When the guest is shut down, the mediated matrix devices may be removed.
+
+Using our example again, to remove the mediated matrix device $uuid1:
+
+ /sys/devices/vfio_ap/matrix/
+ --- [mdev_supported_types]
+ ------ [vfio_ap-passthrough]
+ --------- [devices]
+ ------------ [$uuid1]
+ --------------- remove
+
+
+ echo 1 > remove
+
+ This will remove all of the mdev matrix device's sysfs structures including
+ the mdev device itself. To recreate and reconfigure the mdev matrix device,
+ all of the steps starting with step 3 will have to be performed again. Note
+ that the remove will fail if a guest using the mdev is still running.
+
+ It is not necessary to remove an mdev matrix device, but one may want to
+ remove it if no guest will use it during the remaining lifetime of the linux
+ host. If the mdev matrix device is removed, one may want to also reconfigure
+ the pool of adapters and queues reserved for use by the default drivers.
+
+Limitations
+===========
+* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
+ to the default drivers pool of a queue that is still assigned to a mediated
+ device in use by a guest. It is incumbent upon the administrator to
+ ensure there is no mediated device in use by a guest to which the APQN is
+ assigned lest the host be given access to the private data of the AP queue
+ device such as a private key configured specifically for the guest.
+
+* Dynamically modifying the AP matrix for a running guest (which would amount to
+ hot(un)plug of AP devices for the guest) is currently not supported
+
+* Live guest migration is not supported for guests using AP devices.
diff --git a/MAINTAINERS b/MAINTAINERS
index 5a5aa0f..2dcc26c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12662,6 +12662,7 @@ S: Supported
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
F: drivers/s390/crypto/vfio_ap_ops.c
+F: Documentation/s390/vfio-ap.txt
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.ibm.com>
--
1.7.1
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (25 preceding siblings ...)
2018-09-12 19:43 ` [PATCH v10 26/26] s390: doc: detailed specifications " Tony Krowiak
@ 2018-09-24 10:10 ` Christian Borntraeger
2018-09-24 11:53 ` Cornelia Huck
2018-09-24 16:50 ` Tony Krowiak
2018-09-24 11:49 ` Cornelia Huck
27 siblings, 2 replies; 87+ messages in thread
From: Christian Borntraeger @ 2018-09-24 10:10 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja, Tony Krowiak
FWIW, assuming that review comments for v10 will be addresses in v11, I plan to
add the upcoming v11 to a topic branch and will merge that after 2 or 3 days after
v11 in kvms390/next. In the future the vfio-ap driver will likely be handled by
Martins s390 tree, so I guess it makes sense for him to merge the upcoming
topic branch as well. I will coordinate with Martin.
Christian
On 09/12/2018 09:42 PM, Tony Krowiak wrote:
> From: Tony Krowiak <akrowiak@linux.ibm.com>
>
> Notes:
> =====
>
> Patches 1-2 (by David) are posted with this series because they are not
> currently available in our master branch, upon which this series is based,
> and because this series is dependent upon them.
>
> This patch series works with the v8 QEMU patches.
>
> Abstract:
> ========
>
> On s390, we have cryptographic coprocessor cards, which are modeled on
> Linux as devices on the AP bus. Each card can be partitioned into domains
> which can be thought of as a set of hardware registers for processing
> crypto commands. Crypto commands are sent to a specific domain within a
> card is via a queue which is identified as a (card,domain) tuple. We model
> this something like the following (assuming we have access to cards 3 and
> 4 and domains 1 and 2):
>
> AP -> card3 -> queue (3,1)
> -> queue (3,2)
> -> card4 -> queue (4,1)
> -> queue (4,2)
>
> If we want to virtualize this, we can use a feature provided by the
> hardware. We basically attach a satellite control block to our main
> hardware virtualization control block and the hardware takes care of
> most of the rest.
>
> For this control block, we don't specify explicit tuples, but a list of
> cards and a list of domains. The guest will get access to the cross
> product.
>
> Because of this, we need to take care that the lists provided to
> different guests don't overlap; i.e., we need to enforce sane
> configurations. Otherwise, one guest may get access to things like
> secret keys for another guest.
>
> The idea of this patch set is to introduce a new device, the matrix
> device. This matrix device hangs off a different root and acts as the
> parent node for mdev devices.
>
> If you now want to give the tuples (4,1) and (4,2), you need to do the
> following:
>
> - Make sure the queues (4,1) and (4,2) belong to vfio_ap (see patches
> #5 and #6)
> - Create the mediated device.
> - Assign card 4 and domains 1 and 2 to the mediated device
> - Optionally activate the mediated device.
>
> QEMU will now simply consume the mediated device and things should work.
>
> For a complete description of the architecture and concepts underlying
> the design, see the Documentation/s390/vfio-ap.txt file included with this
> patch set.
>
> v9 => v10 Change log:
> ===================
> * Replaced statically allocated with dynamically allocated matrix device
> * Made changes to drivers/iommu/Kconfig and arch/s390/Kconfig to fix the
> dependency chain so that make menuconfig can be used to configure VFIO_AP
> * Added KVM device attributes to enable/disable hw interpretation of AP
> instructions from userspace.
> * Return more meaningful error values from mediated matrix device
> assignment sysfs interfaces
> * No longer enforcing convention that the ADM is a superset of the AQM at
> for guest level 2
> * Broke 2 vSIE patches into several to make it easier to review them
> * Reworked vSIE patches to handle various CRYCB formats
> v8 => v9 Change log:
> ===================
> * Removed /sys/devices/virtual/misc/vfio_ap device and restored
> /sys/devices/vfio_ap/matrix device as parent of mediated matrix devices
> * Return boolean from ap_configuration_available() in ap.h instead of 0 or
> an error.
> * Miscellaneous changes due to review comments
>
> v7 => v8 Change log:
> ===================
> * Removed the AP bus gained the ability to designate queues as 'used by
> host' or as 'used by alternate driver(s)'.
> * Removed 'activate' attribute from mediated device.
> * Do consistency checking during device assignment:
> 1. Verify that APQNs assigned to the mediated device are bound to the
> VFIO AP device driver
> 2. Verify that no APQN assigned to the mediated matrix device is assigned
> to any other mediated matrix device.
> * The attributes of a mediated matrix device that is in use by a guest can
> not be changed - i.e., no device assignment/unassignment allowed
> * A mediated matrix device that is in use by a guest can not be removed.
> * Removed all printk logging from VFIO AP driver; allowing return codes
> from interfaces to describe the error.
> * Reworked the handling of the CRYCB in vSIE based upon patches introduced
> by David in the mainline.
>
> v6 => v7 Change log:
> ===================
> * The AP bus gained the ability to designate queues as 'used by host'
> or as 'used by alternate driver(s)'. This allows us to authorise access
> (via the CRYCB) to queues that are not currently bound to the vfio_ap
> driver. If a vfio_ap owned queue diss- and reapears it's guaranteed
> to get bound back to the vfio_ap driver.
> * The mediated device gained an 'activate' attribute. Sharing conflicts are
> checked on activation now. If the device was not activated, the mdev
> open still implies activation. An active ap_matrix_mdev device claims
> it's resources -- an inactive does not.
> * An active ap_matrix_mdev device can not be removed. An ap_matrix_mdev
> that is hooked up with a guest can not be deactivated.
> * An active ap_matrix_mdev device rejects assign_* and deassign_*
> operations. Thus changing the CRYCB masks of a guest in order to
> accomplys certain hotplug scenarios is planned, but not supported yet. In
> previous versions it was possible to do those operations on a ap_matrix_mdev
> that is hooked up to a guest, but the changes would take effect on the next
> mdev_open.
> * Synchronisation was reworked.
> * The sysfs path of the parent device changed from /sys/devices/vfio_ap/matrix/
> to /sys/devices/virtual/misc/vfio_ap/. The parent device is a misc
> device now.
> * The severity for most of the messages were reduced form error to
> warning.
> * We are not as thick headed about the zapq as we used to be in v6.
>
> v5 => v6 Change log:
> ===================
> * Added VSIE support - thanks to Pierre Morel
> * Added VFIO_DEVICE_RESET ioctl
> * Zeroizing AP queues when mediated device released and when
> VFIO_DEVICE_RESET ioctl is invoked
> * Removed /arch/s390/kvm/kvm-ap.c and arch/s390/include/asm/kvm-ap.h and
> moved guest matrix configuration into vfio driver
> * Removed temporary interfaces to be supplied by AP bus
> * Made the variable that keeps track of mdev instance count an atomic_t
> type
> * Removed code iterating through vm_list to determine if another guest has
> a queue .... not keep a list of matrix_mdev devices and verify against
> that list. Removes the need for the kvm_lock.
> * Added a sysfs attribute for the mediated matrix device to display the
> matrix contained in the guest's CRYCB if a guest is using the mdev.
>
>
> v4 => v5 Change log:
> ===================
> * Verify AP queues bound to driver in mediated device open callback, prior
> to configuring the matrix in the CRYCB
> * Implement VFIO_DEVICE_RESET ioctl
> * Zeroize queues on guest entry and exit
> * Removed vnet from all email IBM email addresses referenced
> * Add synchronization in mdev create/remove and open/release.
>
> v4 => v5 Change log:
> ===================
> * Added code to mdev open callback to ensure not more than one vfio-ap
> device can be opened by a guest.
> * Interpret AP instructions by default
> * Removed patch implementing interface to enable/disable AP interpretation,
> since that will now be done by default
> * Removed patch to reset crypto attributes for ALL vcpus. That will be
> submitted as a single patch since it will not be needed in this series -
> i.e., it was called from the interface to enable/disable AP instructions
> * All code for initializing crypto for a guest has been moved back to
> kvm-s390.c, kvm_s390_crypto_init(kvm) function
> * Maintaining a module reference count for the vfio_ap module so it is not
> removed while a guest with AP devices is running.
>
>
>
> v3 => v4 Change log:
> ===================
> * Resolved issue with enabling ZCRYPT when KVM is enabled by using
> #ifdef ZCRYPT in relevant functions
> * Added patch with a new function for resetting the crypto attributes
> for all vcpus to resolve the issue raised with running vcpus getting out
> of sync.
> * Removed KVM_S390_VM_CRYPTO_INTERPRET_AP: Setting interpretive exec mode
> from vfio_ap driver when mdev device is opened.
>
>
> v2 => v3 Change log:
> ===================
> * Set APIE in VCPU setup function
> * Renamed patch 13/15:
> KVM: s390: Configure the guest's CRYCB
> KVM: s390: Configure the guest's AP devices
> * Fixed problem with building arch/s390/kvm/kvm-ap.c when CONFIG_ZCRYPT
> not selected
> * Removed patch introducing VSIE support for AP pending further
> investigation
> * Initialized AP maximum mask sizes - i.e., APM, AQM and ADM - from info
> returned from PQAP(QCI) function
> * Introduced a new device attribute to the KVM_S390_VM_CRYPTO attribute
> group for setting a flag via the KVM_SET_DEVICE_ATTR ioctl to indicate
> whether ECA_APIE should be set or not. The flag is used in the
> kvm_s390_vcpu_crypto_setup() function to set ECA_APIE in the SIE block.
>
>
> v1 => v2 Change log:
> ===================
> * Added documentation vfio-ap.txt
> * Renamed vfio_ap_matrix module and device driver to vfio_ap
> * Use device core device list instead of maintaining list of matrix
> devices in driver
> * Added VSIE support for AP
> * Create matrix device before registering VFIO AP device driver with the
> AP bus
> * Renamed the following files in drivers/s390/crypto:
> * vfio_ap_matrix.drv -> vfio_ap_drv
> * vfio_ap_matrix_private.h -> vfio_ap_private.h
> * vfio_ap_matrix_ops.c -> vfio_ap_ops.c
> * arch/s390/include/asm/kvm/ap-matrix-config.h
> * Renamed to kvm-ap.h
> * Changed the data type of the bit mask fields for the matrix structure
> to unsigned long and create them with DECLARE_BITMAP
> * Changed #define prefixes from AP_MATRIX to KVM_AP
> * Changed function and structure prefixes from ap_matrix to kvm_ap
> * Added function interface to check if AP Extended Addressing (APXA)
> facility is installedCRYCB_FORMAT_MASK
> * Added function interface to get the maximum ID for AP mask type
> * Added function interface to set the AP execution mode
> * arch/s390/kvm/ap-matrix-config.c
> * Renamed to kvm-ap.c
> * Changed function prefixes from ap_matrix to kvm_ap
> * Added function to check if AP Extended Addressing (APXA) facility is
> installed
> * Added function to get the maximum ID for AP mask type
> * Added function to set the AP execution mode
> * Added a boolean parameter to the functions that retrieve the APM, AQM
> and ADM bit mask fields from the CRYCB. If true, then the function
> will clear the bits in the mask before returning a reference to it
> * Added validation to verify that APM, AQM and ADM bits that are set do
> not exceed the maximum ID value allowed
> *
> * arch/s390/include/asm/kvm_host.h
> * Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
> * Added a flag to struct kvm_s390_crypto to indicate whether the
> KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
> * Added two CPU facilities features to set STFLE.12 and STFLE.15
> * arch/s390/kvm/kvm-s390.c
> * Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
> * Removed kvm_s390_apxa_installed() function
> * Changed call to kvm_s390_apxa_installed() which has been removed to a
> call to new kvm_ap_apxa_installed() function.
> * Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
> feature flag in the kvm_s390_crypto structure
> * Added CRYCB_FORMAT_MASK to mask CRYCBD
> * arch/s390/tools/gen_facilities.c
> * Added STFLE.12 and STFLE.15 to struct facility _def
> * drivers/s390/crypto/vfio_ap_matrix_private.h
> * Changed name of file to vfio_ap.private.h
> * Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
> * struct ap_matrix: removed list fields and locks
> * struct vfio_ap_queue: removed list field
> * Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
> to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
> * drivers/s390/crypto/vfio_ap_matrix_drv.c
> * Renamed file to drivers/s390/crypto/vfio_ap_drv.c
> * Changed all #define, structure and function prefixes to vfio_ap
> * probe function
> * Changed root device name for the matrix device to vfio_ap:
> i.e., /sys/devices/vfio_ap/matrix
> * No longer storing the AP queue device in a list, it is retrievable via
> the device core
> * Removed unnecessary check whether matrix device exists
> * Store the vfio_ap_queue structure in the private field of the ap_queue
> structure rather than using list interface
> * remove function
> * Retrieve vfio_ap_queue structure from the struct ap_queue private
> data rather than from a list
> * Removed unnecesary check
> * drivers/s390/crypto/vfio_ap_matrix_ops.c
> * Renamed file to vfio_ap_ops.c
> * Changed #define prefixes from AP_MATRIX to VFIO_AP
> * Changed function name prefixes from ap_matrix to vfio_ap
> * Removed ioctl to configure the CRYCB
> * create function
> * Removed ap_matrix_mdev_find_by_uuid() function - function is provided
> by mdev core
> * Removed available_instances verification, provided by mdev core
> * Removed check to see if mediated device exists, handled by mdev core
> * notifier function
> * Configuring matrix here instead of via ioctl
> * Set interpretive execution mode for all VCPUs
> * Removed R/O attributes to display adapters and domains
> * Added an R/O attribute to display the matrix
> * assign_control_domain mdev attribute:
> * Removed check to see if the domain is installed on the linux host
> * Added check to verify the control domain ID does not exceed the max
> value
> * assign_adapter mdev attribute:
> * Added check to verify the adapter ID does not exceed the max
> value
> * If any APQNs configured for the mediated matrix device that
> have an APID matching the adapter ID being assigned are not
> bound to the vfio_ap device driver then it is assumed that the APQN is
> bound to another driver and assignment will fail
> * assign_domain mdev attribute:
> * Added check to verify the domain ID does not exceed the max
> value
> * If any APQNs configured for the mediated matrix device that
> have an APQI matching the domain ID being assigned are not
> bound to the vfio_ap device driver then it is assumed that the APQN is
> bound to another driver and assignment will fail
> * tools/arch/s390/include/uapi/asm/kvm.h
> * removed KVM_S390_VM_CPU_FEAT_AP feature definition
>
> David Hildenbrand (2):
> KVM: s390: vsie: simulate VCPU SIE entry/exit
> KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
>
> Pierre Morel (9):
> KVM: s390: Clear Crypto Control Block when using vSIE
> KVM: s390: vsie: Do the CRYCB validation first
> KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
> KVM: s390: vsie: Allow CRYCB FORMAT-2
> KVM: s390: vsie: allow CRYCB FORMAT-1
> KVM: s390: vsie: allow CRYCB FORMAT-0
> KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
> KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
> KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
>
> Tony Krowiak (15):
> KVM: s390: refactor crypto initialization
> s390: vfio-ap: base implementation of VFIO AP device driver
> s390: vfio-ap: register matrix device with VFIO mdev framework
> s390: vfio-ap: sysfs interfaces to configure adapters
> s390: vfio-ap: sysfs interfaces to configure domains
> s390: vfio-ap: sysfs interfaces to configure control domains
> s390: vfio-ap: sysfs interface to view matrix mdev matrix
> KVM: s390: interfaces to clear CRYCB masks
> s390: vfio-ap: implement mediated device open callback
> s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
> s390: vfio-ap: zeroize the AP queues
> s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
> KVM: s390: device attrs to enable/disable AP interpretation
> KVM: s390: CPU model support for AP virtualization
> s390: doc: detailed specifications for AP virtualization
>
> Documentation/s390/vfio-ap.txt | 778 ++++++++++++++++++++++++++
> MAINTAINERS | 12 +
> arch/s390/Kconfig | 11 +
> arch/s390/include/asm/kvm_host.h | 7 +
> arch/s390/include/uapi/asm/kvm.h | 3 +
> arch/s390/kvm/kvm-s390.c | 134 +++--
> arch/s390/kvm/kvm-s390.h | 1 +
> arch/s390/kvm/vsie.c | 208 +++++++-
> arch/s390/tools/gen_facilities.c | 2 +
> drivers/iommu/Kconfig | 8 +
> drivers/s390/crypto/Makefile | 4 +
> drivers/s390/crypto/vfio_ap_drv.c | 153 ++++++
> drivers/s390/crypto/vfio_ap_ops.c | 973 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 88 +++
> include/uapi/linux/vfio.h | 2 +
> 15 files changed, 2324 insertions(+), 60 deletions(-)
> create mode 100644 Documentation/s390/vfio-ap.txt
> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-24 10:10 ` [PATCH v10 00/26] guest dedicated crypto adapters Christian Borntraeger
@ 2018-09-24 11:53 ` Cornelia Huck
2018-09-24 16:46 ` Tony Krowiak
2018-09-24 16:50 ` Tony Krowiak
1 sibling, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:53 UTC (permalink / raw)
To: Christian Borntraeger
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, kwankhede, bjsdjshi, pbonzini, alex.williamson,
pmorel, alifm, mjrosato, jjherne, thuth, pasic, berrange, fiuczy,
buendgen, frankja, Tony Krowiak
On Mon, 24 Sep 2018 12:10:37 +0200
Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> FWIW, assuming that review comments for v10 will be addresses in v11, I plan to
> add the upcoming v11 to a topic branch and will merge that after 2 or 3 days after
> v11 in kvms390/next.
As said before, I think this looks sane enough to merge with the
requested changes.
> In the future the vfio-ap driver will likely be handled by
> Martins s390 tree, so I guess it makes sense for him to merge the upcoming
> topic branch as well. I will coordinate with Martin.
I assume future patches will still be posted to kvm + linux-s390 for
review?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-24 11:53 ` Cornelia Huck
@ 2018-09-24 16:46 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:46 UTC (permalink / raw)
To: Cornelia Huck, Christian Borntraeger
Cc: Tony Krowiak, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, kwankhede, bjsdjshi, pbonzini, alex.williamson,
pmorel, alifm, mjrosato, jjherne, thuth, pasic, berrange, fiuczy,
buendgen, frankja
On 09/24/2018 07:53 AM, Cornelia Huck wrote:
> On Mon, 24 Sep 2018 12:10:37 +0200
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>
>> FWIW, assuming that review comments for v10 will be addresses in v11, I plan to
>> add the upcoming v11 to a topic branch and will merge that after 2 or 3 days after
>> v11 in kvms390/next.
>
> As said before, I think this looks sane enough to merge with the
> requested changes.
>
>> In the future the vfio-ap driver will likely be handled by
>> Martins s390 tree, so I guess it makes sense for him to merge the upcoming
>> topic branch as well. I will coordinate with Martin.
>
> I assume future patches will still be posted to kvm + linux-s390 for
> review?
Affirmative
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-24 10:10 ` [PATCH v10 00/26] guest dedicated crypto adapters Christian Borntraeger
2018-09-24 11:53 ` Cornelia Huck
@ 2018-09-24 16:50 ` Tony Krowiak
1 sibling, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:50 UTC (permalink / raw)
To: Christian Borntraeger, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, berrange, fiuczy, buendgen, frankja
On 09/24/2018 06:10 AM, Christian Borntraeger wrote:
> FWIW, assuming that review comments for v10 will be addresses in v11, I plan to
> add the upcoming v11 to a topic branch and will merge that after 2 or 3 days after
> v11 in kvms390/next. In the future the vfio-ap driver will likely be handled by
> Martins s390 tree, so I guess it makes sense for him to merge the upcoming
> topic branch as well. I will coordinate with Martin.
>
> Christian
As soon as I see that review comments have stopped coming, I'll post v11
.... probably later this week. It will include all changes accepted from
review.
>
>
> On 09/12/2018 09:42 PM, Tony Krowiak wrote:
>> From: Tony Krowiak <akrowiak@linux.ibm.com>
>>
>> Notes:
>> =====
>>
>> Patches 1-2 (by David) are posted with this series because they are not
>> currently available in our master branch, upon which this series is based,
>> and because this series is dependent upon them.
>>
>> This patch series works with the v8 QEMU patches.
>>
>> Abstract:
>> ========
>>
>> On s390, we have cryptographic coprocessor cards, which are modeled on
>> Linux as devices on the AP bus. Each card can be partitioned into domains
>> which can be thought of as a set of hardware registers for processing
>> crypto commands. Crypto commands are sent to a specific domain within a
>> card is via a queue which is identified as a (card,domain) tuple. We model
>> this something like the following (assuming we have access to cards 3 and
>> 4 and domains 1 and 2):
>>
>> AP -> card3 -> queue (3,1)
>> -> queue (3,2)
>> -> card4 -> queue (4,1)
>> -> queue (4,2)
>>
>> If we want to virtualize this, we can use a feature provided by the
>> hardware. We basically attach a satellite control block to our main
>> hardware virtualization control block and the hardware takes care of
>> most of the rest.
>>
>> For this control block, we don't specify explicit tuples, but a list of
>> cards and a list of domains. The guest will get access to the cross
>> product.
>>
>> Because of this, we need to take care that the lists provided to
>> different guests don't overlap; i.e., we need to enforce sane
>> configurations. Otherwise, one guest may get access to things like
>> secret keys for another guest.
>>
>> The idea of this patch set is to introduce a new device, the matrix
>> device. This matrix device hangs off a different root and acts as the
>> parent node for mdev devices.
>>
>> If you now want to give the tuples (4,1) and (4,2), you need to do the
>> following:
>>
>> - Make sure the queues (4,1) and (4,2) belong to vfio_ap (see patches
>> #5 and #6)
>> - Create the mediated device.
>> - Assign card 4 and domains 1 and 2 to the mediated device
>> - Optionally activate the mediated device.
>>
>> QEMU will now simply consume the mediated device and things should work.
>>
>> For a complete description of the architecture and concepts underlying
>> the design, see the Documentation/s390/vfio-ap.txt file included with this
>> patch set.
>>
>> v9 => v10 Change log:
>> ===================
>> * Replaced statically allocated with dynamically allocated matrix device
>> * Made changes to drivers/iommu/Kconfig and arch/s390/Kconfig to fix the
>> dependency chain so that make menuconfig can be used to configure VFIO_AP
>> * Added KVM device attributes to enable/disable hw interpretation of AP
>> instructions from userspace.
>> * Return more meaningful error values from mediated matrix device
>> assignment sysfs interfaces
>> * No longer enforcing convention that the ADM is a superset of the AQM at
>> for guest level 2
>> * Broke 2 vSIE patches into several to make it easier to review them
>> * Reworked vSIE patches to handle various CRYCB formats
>> v8 => v9 Change log:
>> ===================
>> * Removed /sys/devices/virtual/misc/vfio_ap device and restored
>> /sys/devices/vfio_ap/matrix device as parent of mediated matrix devices
>> * Return boolean from ap_configuration_available() in ap.h instead of 0 or
>> an error.
>> * Miscellaneous changes due to review comments
>>
>> v7 => v8 Change log:
>> ===================
>> * Removed the AP bus gained the ability to designate queues as 'used by
>> host' or as 'used by alternate driver(s)'.
>> * Removed 'activate' attribute from mediated device.
>> * Do consistency checking during device assignment:
>> 1. Verify that APQNs assigned to the mediated device are bound to the
>> VFIO AP device driver
>> 2. Verify that no APQN assigned to the mediated matrix device is assigned
>> to any other mediated matrix device.
>> * The attributes of a mediated matrix device that is in use by a guest can
>> not be changed - i.e., no device assignment/unassignment allowed
>> * A mediated matrix device that is in use by a guest can not be removed.
>> * Removed all printk logging from VFIO AP driver; allowing return codes
>> from interfaces to describe the error.
>> * Reworked the handling of the CRYCB in vSIE based upon patches introduced
>> by David in the mainline.
>>
>> v6 => v7 Change log:
>> ===================
>> * The AP bus gained the ability to designate queues as 'used by host'
>> or as 'used by alternate driver(s)'. This allows us to authorise access
>> (via the CRYCB) to queues that are not currently bound to the vfio_ap
>> driver. If a vfio_ap owned queue diss- and reapears it's guaranteed
>> to get bound back to the vfio_ap driver.
>> * The mediated device gained an 'activate' attribute. Sharing conflicts are
>> checked on activation now. If the device was not activated, the mdev
>> open still implies activation. An active ap_matrix_mdev device claims
>> it's resources -- an inactive does not.
>> * An active ap_matrix_mdev device can not be removed. An ap_matrix_mdev
>> that is hooked up with a guest can not be deactivated.
>> * An active ap_matrix_mdev device rejects assign_* and deassign_*
>> operations. Thus changing the CRYCB masks of a guest in order to
>> accomplys certain hotplug scenarios is planned, but not supported yet. In
>> previous versions it was possible to do those operations on a ap_matrix_mdev
>> that is hooked up to a guest, but the changes would take effect on the next
>> mdev_open.
>> * Synchronisation was reworked.
>> * The sysfs path of the parent device changed from /sys/devices/vfio_ap/matrix/
>> to /sys/devices/virtual/misc/vfio_ap/. The parent device is a misc
>> device now.
>> * The severity for most of the messages were reduced form error to
>> warning.
>> * We are not as thick headed about the zapq as we used to be in v6.
>>
>> v5 => v6 Change log:
>> ===================
>> * Added VSIE support - thanks to Pierre Morel
>> * Added VFIO_DEVICE_RESET ioctl
>> * Zeroizing AP queues when mediated device released and when
>> VFIO_DEVICE_RESET ioctl is invoked
>> * Removed /arch/s390/kvm/kvm-ap.c and arch/s390/include/asm/kvm-ap.h and
>> moved guest matrix configuration into vfio driver
>> * Removed temporary interfaces to be supplied by AP bus
>> * Made the variable that keeps track of mdev instance count an atomic_t
>> type
>> * Removed code iterating through vm_list to determine if another guest has
>> a queue .... not keep a list of matrix_mdev devices and verify against
>> that list. Removes the need for the kvm_lock.
>> * Added a sysfs attribute for the mediated matrix device to display the
>> matrix contained in the guest's CRYCB if a guest is using the mdev.
>>
>>
>> v4 => v5 Change log:
>> ===================
>> * Verify AP queues bound to driver in mediated device open callback, prior
>> to configuring the matrix in the CRYCB
>> * Implement VFIO_DEVICE_RESET ioctl
>> * Zeroize queues on guest entry and exit
>> * Removed vnet from all email IBM email addresses referenced
>> * Add synchronization in mdev create/remove and open/release.
>>
>> v4 => v5 Change log:
>> ===================
>> * Added code to mdev open callback to ensure not more than one vfio-ap
>> device can be opened by a guest.
>> * Interpret AP instructions by default
>> * Removed patch implementing interface to enable/disable AP interpretation,
>> since that will now be done by default
>> * Removed patch to reset crypto attributes for ALL vcpus. That will be
>> submitted as a single patch since it will not be needed in this series -
>> i.e., it was called from the interface to enable/disable AP instructions
>> * All code for initializing crypto for a guest has been moved back to
>> kvm-s390.c, kvm_s390_crypto_init(kvm) function
>> * Maintaining a module reference count for the vfio_ap module so it is not
>> removed while a guest with AP devices is running.
>>
>>
>>
>> v3 => v4 Change log:
>> ===================
>> * Resolved issue with enabling ZCRYPT when KVM is enabled by using
>> #ifdef ZCRYPT in relevant functions
>> * Added patch with a new function for resetting the crypto attributes
>> for all vcpus to resolve the issue raised with running vcpus getting out
>> of sync.
>> * Removed KVM_S390_VM_CRYPTO_INTERPRET_AP: Setting interpretive exec mode
>> from vfio_ap driver when mdev device is opened.
>>
>>
>> v2 => v3 Change log:
>> ===================
>> * Set APIE in VCPU setup function
>> * Renamed patch 13/15:
>> KVM: s390: Configure the guest's CRYCB
>> KVM: s390: Configure the guest's AP devices
>> * Fixed problem with building arch/s390/kvm/kvm-ap.c when CONFIG_ZCRYPT
>> not selected
>> * Removed patch introducing VSIE support for AP pending further
>> investigation
>> * Initialized AP maximum mask sizes - i.e., APM, AQM and ADM - from info
>> returned from PQAP(QCI) function
>> * Introduced a new device attribute to the KVM_S390_VM_CRYPTO attribute
>> group for setting a flag via the KVM_SET_DEVICE_ATTR ioctl to indicate
>> whether ECA_APIE should be set or not. The flag is used in the
>> kvm_s390_vcpu_crypto_setup() function to set ECA_APIE in the SIE block.
>>
>>
>> v1 => v2 Change log:
>> ===================
>> * Added documentation vfio-ap.txt
>> * Renamed vfio_ap_matrix module and device driver to vfio_ap
>> * Use device core device list instead of maintaining list of matrix
>> devices in driver
>> * Added VSIE support for AP
>> * Create matrix device before registering VFIO AP device driver with the
>> AP bus
>> * Renamed the following files in drivers/s390/crypto:
>> * vfio_ap_matrix.drv -> vfio_ap_drv
>> * vfio_ap_matrix_private.h -> vfio_ap_private.h
>> * vfio_ap_matrix_ops.c -> vfio_ap_ops.c
>> * arch/s390/include/asm/kvm/ap-matrix-config.h
>> * Renamed to kvm-ap.h
>> * Changed the data type of the bit mask fields for the matrix structure
>> to unsigned long and create them with DECLARE_BITMAP
>> * Changed #define prefixes from AP_MATRIX to KVM_AP
>> * Changed function and structure prefixes from ap_matrix to kvm_ap
>> * Added function interface to check if AP Extended Addressing (APXA)
>> facility is installedCRYCB_FORMAT_MASK
>> * Added function interface to get the maximum ID for AP mask type
>> * Added function interface to set the AP execution mode
>> * arch/s390/kvm/ap-matrix-config.c
>> * Renamed to kvm-ap.c
>> * Changed function prefixes from ap_matrix to kvm_ap
>> * Added function to check if AP Extended Addressing (APXA) facility is
>> installed
>> * Added function to get the maximum ID for AP mask type
>> * Added function to set the AP execution mode
>> * Added a boolean parameter to the functions that retrieve the APM, AQM
>> and ADM bit mask fields from the CRYCB. If true, then the function
>> will clear the bits in the mask before returning a reference to it
>> * Added validation to verify that APM, AQM and ADM bits that are set do
>> not exceed the maximum ID value allowed
>> *
>> * arch/s390/include/asm/kvm_host.h
>> * Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
>> * Added a flag to struct kvm_s390_crypto to indicate whether the
>> KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
>> * Added two CPU facilities features to set STFLE.12 and STFLE.15
>> * arch/s390/kvm/kvm-s390.c
>> * Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
>> * Removed kvm_s390_apxa_installed() function
>> * Changed call to kvm_s390_apxa_installed() which has been removed to a
>> call to new kvm_ap_apxa_installed() function.
>> * Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
>> feature flag in the kvm_s390_crypto structure
>> * Added CRYCB_FORMAT_MASK to mask CRYCBD
>> * arch/s390/tools/gen_facilities.c
>> * Added STFLE.12 and STFLE.15 to struct facility _def
>> * drivers/s390/crypto/vfio_ap_matrix_private.h
>> * Changed name of file to vfio_ap.private.h
>> * Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
>> * struct ap_matrix: removed list fields and locks
>> * struct vfio_ap_queue: removed list field
>> * Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
>> to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
>> * drivers/s390/crypto/vfio_ap_matrix_drv.c
>> * Renamed file to drivers/s390/crypto/vfio_ap_drv.c
>> * Changed all #define, structure and function prefixes to vfio_ap
>> * probe function
>> * Changed root device name for the matrix device to vfio_ap:
>> i.e., /sys/devices/vfio_ap/matrix
>> * No longer storing the AP queue device in a list, it is retrievable via
>> the device core
>> * Removed unnecessary check whether matrix device exists
>> * Store the vfio_ap_queue structure in the private field of the ap_queue
>> structure rather than using list interface
>> * remove function
>> * Retrieve vfio_ap_queue structure from the struct ap_queue private
>> data rather than from a list
>> * Removed unnecesary check
>> * drivers/s390/crypto/vfio_ap_matrix_ops.c
>> * Renamed file to vfio_ap_ops.c
>> * Changed #define prefixes from AP_MATRIX to VFIO_AP
>> * Changed function name prefixes from ap_matrix to vfio_ap
>> * Removed ioctl to configure the CRYCB
>> * create function
>> * Removed ap_matrix_mdev_find_by_uuid() function - function is provided
>> by mdev core
>> * Removed available_instances verification, provided by mdev core
>> * Removed check to see if mediated device exists, handled by mdev core
>> * notifier function
>> * Configuring matrix here instead of via ioctl
>> * Set interpretive execution mode for all VCPUs
>> * Removed R/O attributes to display adapters and domains
>> * Added an R/O attribute to display the matrix
>> * assign_control_domain mdev attribute:
>> * Removed check to see if the domain is installed on the linux host
>> * Added check to verify the control domain ID does not exceed the max
>> value
>> * assign_adapter mdev attribute:
>> * Added check to verify the adapter ID does not exceed the max
>> value
>> * If any APQNs configured for the mediated matrix device that
>> have an APID matching the adapter ID being assigned are not
>> bound to the vfio_ap device driver then it is assumed that the APQN is
>> bound to another driver and assignment will fail
>> * assign_domain mdev attribute:
>> * Added check to verify the domain ID does not exceed the max
>> value
>> * If any APQNs configured for the mediated matrix device that
>> have an APQI matching the domain ID being assigned are not
>> bound to the vfio_ap device driver then it is assumed that the APQN is
>> bound to another driver and assignment will fail
>> * tools/arch/s390/include/uapi/asm/kvm.h
>> * removed KVM_S390_VM_CPU_FEAT_AP feature definition
>>
>> David Hildenbrand (2):
>> KVM: s390: vsie: simulate VCPU SIE entry/exit
>> KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
>>
>> Pierre Morel (9):
>> KVM: s390: Clear Crypto Control Block when using vSIE
>> KVM: s390: vsie: Do the CRYCB validation first
>> KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
>> KVM: s390: vsie: Allow CRYCB FORMAT-2
>> KVM: s390: vsie: allow CRYCB FORMAT-1
>> KVM: s390: vsie: allow CRYCB FORMAT-0
>> KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
>> KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
>> KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
>>
>> Tony Krowiak (15):
>> KVM: s390: refactor crypto initialization
>> s390: vfio-ap: base implementation of VFIO AP device driver
>> s390: vfio-ap: register matrix device with VFIO mdev framework
>> s390: vfio-ap: sysfs interfaces to configure adapters
>> s390: vfio-ap: sysfs interfaces to configure domains
>> s390: vfio-ap: sysfs interfaces to configure control domains
>> s390: vfio-ap: sysfs interface to view matrix mdev matrix
>> KVM: s390: interfaces to clear CRYCB masks
>> s390: vfio-ap: implement mediated device open callback
>> s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
>> s390: vfio-ap: zeroize the AP queues
>> s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
>> KVM: s390: device attrs to enable/disable AP interpretation
>> KVM: s390: CPU model support for AP virtualization
>> s390: doc: detailed specifications for AP virtualization
>>
>> Documentation/s390/vfio-ap.txt | 778 ++++++++++++++++++++++++++
>> MAINTAINERS | 12 +
>> arch/s390/Kconfig | 11 +
>> arch/s390/include/asm/kvm_host.h | 7 +
>> arch/s390/include/uapi/asm/kvm.h | 3 +
>> arch/s390/kvm/kvm-s390.c | 134 +++--
>> arch/s390/kvm/kvm-s390.h | 1 +
>> arch/s390/kvm/vsie.c | 208 +++++++-
>> arch/s390/tools/gen_facilities.c | 2 +
>> drivers/iommu/Kconfig | 8 +
>> drivers/s390/crypto/Makefile | 4 +
>> drivers/s390/crypto/vfio_ap_drv.c | 153 ++++++
>> drivers/s390/crypto/vfio_ap_ops.c | 973 +++++++++++++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_private.h | 88 +++
>> include/uapi/linux/vfio.h | 2 +
>> 15 files changed, 2324 insertions(+), 60 deletions(-)
>> create mode 100644 Documentation/s390/vfio-ap.txt
>> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-12 19:42 [PATCH v10 00/26] guest dedicated crypto adapters Tony Krowiak
` (26 preceding siblings ...)
2018-09-24 10:10 ` [PATCH v10 00/26] guest dedicated crypto adapters Christian Borntraeger
@ 2018-09-24 11:49 ` Cornelia Huck
2018-09-24 16:45 ` Tony Krowiak
27 siblings, 1 reply; 87+ messages in thread
From: Cornelia Huck @ 2018-09-24 11:49 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja, Tony Krowiak
On Wed, 12 Sep 2018 15:42:50 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> Abstract:
> ========
>
> On s390, we have cryptographic coprocessor cards, which are modeled on
> Linux as devices on the AP bus. Each card can be partitioned into domains
> which can be thought of as a set of hardware registers for processing
> crypto commands. Crypto commands are sent to a specific domain within a
> card is via a queue which is identified as a (card,domain) tuple. We model
> this something like the following (assuming we have access to cards 3 and
> 4 and domains 1 and 2):
>
> AP -> card3 -> queue (3,1)
> -> queue (3,2)
> -> card4 -> queue (4,1)
> -> queue (4,2)
>
> If we want to virtualize this, we can use a feature provided by the
> hardware. We basically attach a satellite control block to our main
> hardware virtualization control block and the hardware takes care of
> most of the rest.
>
> For this control block, we don't specify explicit tuples, but a list of
> cards and a list of domains. The guest will get access to the cross
> product.
>
> Because of this, we need to take care that the lists provided to
> different guests don't overlap; i.e., we need to enforce sane
> configurations. Otherwise, one guest may get access to things like
> secret keys for another guest.
>
> The idea of this patch set is to introduce a new device, the matrix
> device. This matrix device hangs off a different root and acts as the
> parent node for mdev devices.
>
> If you now want to give the tuples (4,1) and (4,2), you need to do the
> following:
>
> - Make sure the queues (4,1) and (4,2) belong to vfio_ap (see patches
> #5 and #6)
> - Create the mediated device.
> - Assign card 4 and domains 1 and 2 to the mediated device
> - Optionally activate the mediated device.
>
> QEMU will now simply consume the mediated device and things should work.
>
> For a complete description of the architecture and concepts underlying
> the design, see the Documentation/s390/vfio-ap.txt file included with this
> patch set.
I did not spot anything major, and if v11 addresses the issues raised
by various reviewers I don't see why it should not be merged (interface
looks sane). I skipped looking at the vsie stuff, though ;)
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v10 00/26] guest dedicated crypto adapters
2018-09-24 11:49 ` Cornelia Huck
@ 2018-09-24 16:45 ` Tony Krowiak
0 siblings, 0 replies; 87+ messages in thread
From: Tony Krowiak @ 2018-09-24 16:45 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
berrange, fiuczy, buendgen, frankja
On 09/24/2018 07:49 AM, Cornelia Huck wrote:
> On Wed, 12 Sep 2018 15:42:50 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Abstract:
>> ========
>>
>> On s390, we have cryptographic coprocessor cards, which are modeled on
>> Linux as devices on the AP bus. Each card can be partitioned into domains
>> which can be thought of as a set of hardware registers for processing
>> crypto commands. Crypto commands are sent to a specific domain within a
>> card is via a queue which is identified as a (card,domain) tuple. We model
>> this something like the following (assuming we have access to cards 3 and
>> 4 and domains 1 and 2):
>>
>> AP -> card3 -> queue (3,1)
>> -> queue (3,2)
>> -> card4 -> queue (4,1)
>> -> queue (4,2)
>>
>> If we want to virtualize this, we can use a feature provided by the
>> hardware. We basically attach a satellite control block to our main
>> hardware virtualization control block and the hardware takes care of
>> most of the rest.
>>
>> For this control block, we don't specify explicit tuples, but a list of
>> cards and a list of domains. The guest will get access to the cross
>> product.
>>
>> Because of this, we need to take care that the lists provided to
>> different guests don't overlap; i.e., we need to enforce sane
>> configurations. Otherwise, one guest may get access to things like
>> secret keys for another guest.
>>
>> The idea of this patch set is to introduce a new device, the matrix
>> device. This matrix device hangs off a different root and acts as the
>> parent node for mdev devices.
>>
>> If you now want to give the tuples (4,1) and (4,2), you need to do the
>> following:
>>
>> - Make sure the queues (4,1) and (4,2) belong to vfio_ap (see patches
>> #5 and #6)
>> - Create the mediated device.
>> - Assign card 4 and domains 1 and 2 to the mediated device
>> - Optionally activate the mediated device.
>>
>> QEMU will now simply consume the mediated device and things should work.
>>
>> For a complete description of the architecture and concepts underlying
>> the design, see the Documentation/s390/vfio-ap.txt file included with this
>> patch set.
>
> I did not spot anything major, and if v11 addresses the issues raised
> by various reviewers I don't see why it should not be merged (interface
> looks sane). I skipped looking at the vsie stuff, though ;)
I think David has that covered.
>
^ permalink raw reply [flat|nested] 87+ messages in thread