[Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Peter Maydell]

I'm working on turning on EL2 support in our TCG ARM emulation,
and one area I'm not sure about is whether it should default to
on or off.

We have a few precedents:

For EL3 support:
 * the CPU property is enabled by default but can be disabled by the board
 * 'virt' defaults it to disabled, with a -machine secure=on property
   which turns it on (barfing if KVM is enabled) and also does some other
   stuff like wiring up secure-only memory and devices and making sure
   the GIC has security enabled
 * if the user does -cpu has_el3=yes things will probably not go
   too well and that's unsupported

For PMU support:
 * the CPU property is enabled by default
 * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
 * we do expect the user to be able to turn it on and off with
   a -cpu pmu=yes/no option (and the board model changes behaviour
   based on whether the CPU claims to have the feature)

So what about EL2? Some background facts that are probably relevant:
 * at the moment KVM can't support it, but eventually machines with
   the nested virtualization hardware support will appear (this is
   an ARMv8.3 feature), at which point it ought to work there too
 * if you just enable EL2 then some currently working setups stop
   working (basically any code that was written to assume it was
   entered in EL1); notably this includes kvm-unit-tests (patches
   pending from Drew that fix that). Linux is fine though as it
   checks and handles both.

Disabled-by-default has the advantage that (a) a command line
will work the same for TCG and KVM and (b) nothing that used to
work will break. The disadvantage is that anybody who wants to
be able to run nested KVM will now have to specify a command line
option of some kind.

So:
 (1) should we default on or off? (both at the board level,
     and for the cpu object itself)
 (2) directly user-set cpu property, or board property ?

thanks
-- PMM

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Edgar E. Iglesias]

On Tue, Nov 01, 2016 at 05:16:59PM +0000, Peter Maydell wrote:
> I'm working on turning on EL2 support in our TCG ARM emulation,
> and one area I'm not sure about is whether it should default to
> on or off.
> 
> We have a few precedents:
> 
> For EL3 support:
>  * the CPU property is enabled by default but can be disabled by the board
>  * 'virt' defaults it to disabled, with a -machine secure=on property
>    which turns it on (barfing if KVM is enabled) and also does some other
>    stuff like wiring up secure-only memory and devices and making sure
>    the GIC has security enabled
>  * if the user does -cpu has_el3=yes things will probably not go
>    too well and that's unsupported
> 
> For PMU support:
>  * the CPU property is enabled by default
>  * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
>  * we do expect the user to be able to turn it on and off with
>    a -cpu pmu=yes/no option (and the board model changes behaviour
>    based on whether the CPU claims to have the feature)
> 
> So what about EL2? Some background facts that are probably relevant:
>  * at the moment KVM can't support it, but eventually machines with
>    the nested virtualization hardware support will appear (this is
>    an ARMv8.3 feature), at which point it ought to work there too
>  * if you just enable EL2 then some currently working setups stop
>    working (basically any code that was written to assume it was
>    entered in EL1); notably this includes kvm-unit-tests (patches
>    pending from Drew that fix that). Linux is fine though as it
>    checks and handles both.
> 
> Disabled-by-default has the advantage that (a) a command line
> will work the same for TCG and KVM and (b) nothing that used to
> work will break. The disadvantage is that anybody who wants to
> be able to run nested KVM will now have to specify a command line
> option of some kind.
> 
> So:
>  (1) should we default on or off? (both at the board level,
>      and for the cpu object itself)
>  (2) directly user-set cpu property, or board property ?


Hi Peter,

Good questions...

I wouldn't mind having a board level prop that defaults to off to keep
things somewhat consistent with current KVM and also the secure/EL3 props.

Best regards,
Edgar

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Andrew Jones]

On Tue, Nov 01, 2016 at 05:16:59PM +0000, Peter Maydell wrote:
> I'm working on turning on EL2 support in our TCG ARM emulation,
> and one area I'm not sure about is whether it should default to
> on or off.
> 
> We have a few precedents:
> 
> For EL3 support:
>  * the CPU property is enabled by default but can be disabled by the board
>  * 'virt' defaults it to disabled, with a -machine secure=on property
>    which turns it on (barfing if KVM is enabled) and also does some other
>    stuff like wiring up secure-only memory and devices and making sure
>    the GIC has security enabled

Looks like EL3 support requires enough board changes that it's really
a board feature that requires cpu support (EL3)

>  * if the user does -cpu has_el3=yes things will probably not go
>    too well and that's unsupported

And this confirms that it's a board feature that requires a cpu
feature to be enabled, because just enabling the cpu feature alone
is not only insufficient, but harmful.

(I'll go off-topic here and state that properties like has_el3, which
 we don't want users setting, probably shouldn't be user-settable.
 This has come up a few times before and I think Igor once proposed
 to extend QOM properties to have a "user visible" property that could
 be toggled.)

> 
> For PMU support:
>  * the CPU property is enabled by default
>  * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
>  * we do expect the user to be able to turn it on and off with
>    a -cpu pmu=yes/no option (and the board model changes behaviour
>    based on whether the CPU claims to have the feature)

So here we have the opposite; we have a cpu feature that requires
changes to the board (assignment of an IRQ) to make it more useful.
But, if we used -cpu pmu=on with a board that doesn't assign an IRQ,
then that's OK, the PMU just isn't as useful. So this is definitely
a cpu feature, thus the user-settable cpu property (and lack of a
board property), makes sense.

> 
> So what about EL2? Some background facts that are probably relevant:
>  * at the moment KVM can't support it, but eventually machines with
>    the nested virtualization hardware support will appear (this is
>    an ARMv8.3 feature), at which point it ought to work there too
>  * if you just enable EL2 then some currently working setups stop
>    working (basically any code that was written to assume it was
>    entered in EL1); notably this includes kvm-unit-tests (patches
>    pending from Drew that fix that). Linux is fine though as it
>    checks and handles both.

Have we checked the status of AAVMF yet?

> 
> Disabled-by-default has the advantage that (a) a command line
> will work the same for TCG and KVM and (b) nothing that used to
> work will break. The disadvantage is that anybody who wants to
> be able to run nested KVM will now have to specify a command line
> option of some kind.
> 
> So:
>  (1) should we default on or off? (both at the board level,
>      and for the cpu object itself)

I vote off by default for the two reasons (a and b) you stated
above and also because starting in EL2 may trigger different paths
for the guest, ones that aren't necessary for the "normal" non-
virt enabling use cases. Linux probably won't be adversely 
affected by this, but maybe other guest types would. We'd want
the default to match what we expect to be "normal" use.

>  (2) directly user-set cpu property, or board property ?

So this is why I was rambling above about the two precedents.
IMO, we should choose a cpu property when it's a purely cpu
feature, like the PMU, and board (without a user-settable cpu
property) otherwise. Does this feature require board changes?
I'm aware of switching the psci method from hvc to smc, but
I'd shrug that one off. Is there anything else? If not, then
I vote cpu property only.

Thanks,
drew

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Laszlo Ersek]

On 11/01/16 18:16, Peter Maydell wrote:
> I'm working on turning on EL2 support in our TCG ARM emulation,
> and one area I'm not sure about is whether it should default to
> on or off.
> 
> We have a few precedents:
> 
> For EL3 support:
>  * the CPU property is enabled by default but can be disabled by the board
>  * 'virt' defaults it to disabled, with a -machine secure=on property
>    which turns it on (barfing if KVM is enabled) and also does some other
>    stuff like wiring up secure-only memory and devices and making sure
>    the GIC has security enabled
>  * if the user does -cpu has_el3=yes things will probably not go
>    too well and that's unsupported
> 
> For PMU support:
>  * the CPU property is enabled by default
>  * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
>  * we do expect the user to be able to turn it on and off with
>    a -cpu pmu=yes/no option (and the board model changes behaviour
>    based on whether the CPU claims to have the feature)
> 
> So what about EL2? Some background facts that are probably relevant:
>  * at the moment KVM can't support it, but eventually machines with
>    the nested virtualization hardware support will appear (this is
>    an ARMv8.3 feature), at which point it ought to work there too
>  * if you just enable EL2 then some currently working setups stop
>    working (basically any code that was written to assume it was
>    entered in EL1); notably this includes kvm-unit-tests (patches
>    pending from Drew that fix that). Linux is fine though as it
>    checks and handles both.
> 
> Disabled-by-default has the advantage that (a) a command line
> will work the same for TCG and KVM

Definitely a bonus in my book.

> and (b) nothing that used to
> work will break.

I consider this a requirement, sort of. Regressions are very very
annoying (unless we can prove the previous command line was bogus to
begin with, and just happened to work -- I don't think that applies in
this case).

> The disadvantage is that anybody who wants to
> be able to run nested KVM will now have to specify a command line
> option of some kind.

Nested KVM is sophisticated / non-standard enough that one more tweak
for utilizing it shouldn't be a problem. For example, considering the
kvm_intel host module (yes, it's x86, but the parallel is obvious), it
has an explicit parameter called "nested", which defaults to N.

arch/x86/kvm/vmx.c:

/*
 * If nested=1, nested virtualization is supported, i.e., guests may use
 * VMX and be a hypervisor for its own guests. If nested=0, guests may not
 * use VMX instructions.
 */
static bool __read_mostly nested = 0;
module_param(nested, bool, S_IRUGO);

Additionally, even with

  modprobe kvm_intel nested=1

I think the explicit setting

  -cpu MODEL,+vmx

remains necessary.

Thus, "disabled by default", please.

Thanks
Laszlo

> 
> So:
>  (1) should we default on or off? (both at the board level,
>      and for the cpu object itself)
>  (2) directly user-set cpu property, or board property ?
> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Alex Bennée]

Laszlo Ersek <lersek@redhat.com> writes:

> On 11/01/16 18:16, Peter Maydell wrote:
>> I'm working on turning on EL2 support in our TCG ARM emulation,
>> and one area I'm not sure about is whether it should default to
>> on or off.
>>
>> We have a few precedents:
>>
>> For EL3 support:
>>  * the CPU property is enabled by default but can be disabled by the board
>>  * 'virt' defaults it to disabled, with a -machine secure=on property
>>    which turns it on (barfing if KVM is enabled) and also does some other
>>    stuff like wiring up secure-only memory and devices and making sure
>>    the GIC has security enabled
>>  * if the user does -cpu has_el3=yes things will probably not go
>>    too well and that's unsupported
>>
>> For PMU support:
>>  * the CPU property is enabled by default
>>  * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
>>  * we do expect the user to be able to turn it on and off with
>>    a -cpu pmu=yes/no option (and the board model changes behaviour
>>    based on whether the CPU claims to have the feature)
>>
>> So what about EL2? Some background facts that are probably relevant:
>>  * at the moment KVM can't support it, but eventually machines with
>>    the nested virtualization hardware support will appear (this is
>>    an ARMv8.3 feature), at which point it ought to work there too
>>  * if you just enable EL2 then some currently working setups stop
>>    working (basically any code that was written to assume it was
>>    entered in EL1); notably this includes kvm-unit-tests (patches
>>    pending from Drew that fix that). Linux is fine though as it
>>    checks and handles both.
>>
>> Disabled-by-default has the advantage that (a) a command line
>> will work the same for TCG and KVM
>
> Definitely a bonus in my book.
>
>> and (b) nothing that used to
>> work will break.
>
> I consider this a requirement, sort of. Regressions are very very
> annoying (unless we can prove the previous command line was bogus to
> begin with, and just happened to work -- I don't think that applies in
> this case).
>
>> The disadvantage is that anybody who wants to
>> be able to run nested KVM will now have to specify a command line
>> option of some kind.
>
> Nested KVM is sophisticated / non-standard enough that one more tweak
> for utilizing it shouldn't be a problem. For example, considering the
> kvm_intel host module (yes, it's x86, but the parallel is obvious), it
> has an explicit parameter called "nested", which defaults to N.

I think if we had a board model other than virt for aarch64 then it
would make sense to behave as close to the real hardware as possible.
However as virt is the default model and is intended for virtualisation
I'd also side with el2 being off by default.

If we ever get an SBSA model then I would expect that to start in as
high a privilege level as it should.

--
Alex Bennée

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Peter Maydell]

On 2 November 2016 at 13:59, Andrew Jones <drjones@redhat.com> wrote:
> On Tue, Nov 01, 2016 at 05:16:59PM +0000, Peter Maydell wrote:
>>  (2) directly user-set cpu property, or board property ?
>
> So this is why I was rambling above about the two precedents.
> IMO, we should choose a cpu property when it's a purely cpu
> feature, like the PMU, and board (without a user-settable cpu
> property) otherwise. Does this feature require board changes?
> I'm aware of switching the psci method from hvc to smc, but
> I'd shrug that one off. Is there anything else? If not, then
> I vote cpu property only.

So the things that the board needs to support EL2:
 * PSCI method change
 * need to wire up some interrupt lines (GICv3 maintenance
   interrupt, and VIRQ and VFIQ)
 * for GICv2 you would need also to set a property on
   the GIC to tell it to implement the virt extensions
   (for GICv3 this isn't necessary as the virt extensions
   only affect the CPU interface, which is strictly part
   of the CPU and which in my patches is configured via
   a back-door connection between the CPU proper and the
   GIC cpuif code)

You can do all the interrupt wiring unconditionally,
because if the CPU doesn't have EL2 then the extra
irq lines are just unused. And of course you can drive
the board changes off a CPU property value if you want...

The other reason I made EL3 have both a CPU property and a
board property is that it allowed us to have the board
property default to false (for back-compat) but the
CPU property default to true (so that any future boards
would get EL3 support by default, like the h/w -- you
can't get a real Cortex-A57 without EL2 and EL3).

thanks
-- PMM

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Andrew Jones]

On Mon, Dec 12, 2016 at 12:48:14PM +0000, Peter Maydell wrote:
> On 2 November 2016 at 13:59, Andrew Jones <drjones@redhat.com> wrote:
> > On Tue, Nov 01, 2016 at 05:16:59PM +0000, Peter Maydell wrote:
> >>  (2) directly user-set cpu property, or board property ?
> >
> > So this is why I was rambling above about the two precedents.
> > IMO, we should choose a cpu property when it's a purely cpu
> > feature, like the PMU, and board (without a user-settable cpu
> > property) otherwise. Does this feature require board changes?
> > I'm aware of switching the psci method from hvc to smc, but
> > I'd shrug that one off. Is there anything else? If not, then
> > I vote cpu property only.
> 
> So the things that the board needs to support EL2:
>  * PSCI method change
>  * need to wire up some interrupt lines (GICv3 maintenance
>    interrupt, and VIRQ and VFIQ)
>  * for GICv2 you would need also to set a property on
>    the GIC to tell it to implement the virt extensions
>    (for GICv3 this isn't necessary as the virt extensions
>    only affect the CPU interface, which is strictly part
>    of the CPU and which in my patches is configured via
>    a back-door connection between the CPU proper and the
>    GIC cpuif code)
> 
> You can do all the interrupt wiring unconditionally,
> because if the CPU doesn't have EL2 then the extra
> irq lines are just unused.

Sounds good.

> And of course you can drive
> the board changes off a CPU property value if you want...

Fair point. What do we typically do? I think it'd be nice to be
consistent here. Do cpu properties typically [silently] drive
board changes? Or do board properties, depending on cpu properties,
simply fail when those cpu properties aren't selected by the user,
or do they silently enable them?

> 
> The other reason I made EL3 have both a CPU property and a
> board property is that it allowed us to have the board
> property default to false (for back-compat) but the
> CPU property default to true (so that any future boards
> would get EL3 support by default, like the h/w -- you
> can't get a real Cortex-A57 without EL2 and EL3).

This makes sense to me. A good goal is to keep the processor model
as close as possible to real hardware. I guess that pretty much
just leaves one option: creating a board property that enables the
there-by-default (but still special) cpu feature use.

So, EL2 cpu property on by default, but EL2 board property off
by default. And with EL2 board property off, El2 cpu property
automatically turned off?

Thanks,
drew

> 
> thanks
> -- PMM

Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Peter Maydell]

On 12 December 2016 at 13:22, Andrew Jones <drjones@redhat.com> wrote:
> On Mon, Dec 12, 2016 at 12:48:14PM +0000, Peter Maydell wrote:
>> And of course you can drive
>> the board changes off a CPU property value if you want...
>
> Fair point. What do we typically do? I think it'd be nice to be
> consistent here.

We're not terribly consistent for ARM. We do this for 'pmu' but
not for EL3. (Part of my motivation for this thread was to
find out which of these two precedents we should follow for
EL2.)

> Do cpu properties typically [silently] drive
> board changes? Or do board properties, depending on cpu properties,
> simply fail when those cpu properties aren't selected by the user,
> or do they silently enable them?

For ARM so far the answer has mostly been "CPU properties are
there for the board to set, either unconditionally or sometimes
based on a board property; the user isn't intended to set them,
and if the user does mess with them results are unsupported".
The PMU property is an exception to this rule (perhaps our only
exception so far, though the "define the SCTLR.EE reset value"
property patches currently on the list would be another where
the user is allowed to set them in some cases).

I'm not sure how other CPU architectures handle this.

>> The other reason I made EL3 have both a CPU property and a
>> board property is that it allowed us to have the board
>> property default to false (for back-compat) but the
>> CPU property default to true (so that any future boards
>> would get EL3 support by default, like the h/w -- you
>> can't get a real Cortex-A57 without EL2 and EL3).
>
> This makes sense to me. A good goal is to keep the processor model
> as close as possible to real hardware. I guess that pretty much
> just leaves one option: creating a board property that enables the
> there-by-default (but still special) cpu feature use.
>
> So, EL2 cpu property on by default, but EL2 board property off
> by default. And with EL2 board property off, El2 cpu property
> automatically turned off?

Yes; this matches how we handle EL3, anyway.

thanks
-- PMM