ARM processor mode, kernel startup, Hyp / secure state

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

We're looking into porting Xen to the ARM A15 architecture.  In this
context, it's necessary to arrange for the Xen hypervisor to be
entered from the boot loader in an appropriate processor mode.

KVM needs to deal with the same problem, of course.  And any future
Linux kernel feature which uses the Secure State does too.

We are currently working with ARM's "Fast Model" of the Cortex A15, a
software emulator.  We're using it in the mode where you feed it an
ELF and it loads it into simulated RAM and starts executing at the
ELF entrypoint.  It does this in the CPU's defined startup mode, which
is Kernel mode in the Secure state.

It seems that this environment is what the nascent KVM-on-A15
developers [1] are using too.  There is modified version of the tiny
boot wrapper (the normal version of which just emulates the proper
calling API for the kernel); it sets up a trampoline security monitor.

We need to define a correct calling convention for the kernel which is
compatible with old systems, but which also allows the booted kernel
(Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of
all the available facilities.  The correct approach does seem to be to
have Linux set itself up a trivial trampoline which allows the kernel
to later regain the elevated privilege.

There are a couple of things with the existing KVM ARM approach with
the trivial boot wrapper which need to be fixed, though: firstly,
there should be separate trampolines for hypervisor mode and for
secure state.  That allows the two features to be used independently.
Secondly, the trivial trampolines should be part of the kernel proper
and their lifetime should not extend across the bootloader interface.

At first I thought that the best thing to do would be to boot the
kernel in any suitable mode, and have the kernel automatically detect
the starting mode.  I started writing code in linux's head.S to do
this.  However, detecting whether we are in secure state is very
difficult: it involves deliberately risking an undefined instruction
trap.  The code for this was getting rather long and involved.

Also, unconditionally starting a kernel in hypervisor mode seems
rather unfriendly.  At the moment we unconditionally start it in the
secure state and indeed in the current setup it seems to run entirely
in secure state.  It seems to me that the kernel should mostly run in
non-secure state.

So I propose the following approach:

 1. The kernel will advertise via ELF notes what modes it may be
    started in.  The possible modes will be:
       (i)   secure monitor mode
       (ii)  non-secure hypervisor mode
       (iii) non-secure kernel mode

 2. The bootloader will select the first mode from the three listed
    above which is supported by both the processor and the kernel to
    be loaded, and transition the processor to that mode.  If this
    involves dropping out of secure or hypervisor mode, it will put
    those modes permanently beyond use.

 3. The kernel will examine CPSR to determine which of the three
    possibilities above has happened, and:
   (a) If started in monitor mode:
       * Grant access to everything to non-secure state
       * Set the non-secure copies of the various CP15 registers
         which don't have a sane value on cpu reset
       * Install a trivial monitor vector which unconditionally
         copies r0 to MVBAR and returns
       * Switch to non-secure Hyp mode (if available) and do (b),
         or non-secure Kernel mode (if Hyp mode not supported).         
   (b) If started in Hyp mode:
       * Install a trivial hypervisor vector which unconditionally
         copies r0 to HVBAR and returns
   (c) Rest of startup.

Questions:

 1. What do people think ?  If this seems plausible I will prepare
    an RFC patch for Linux and the boot-wrapper.git.

 2. This cpu startup process must happen very early - before paging
    is enabled, in any case, so before RAM is really available.
    However, it produces two bits of information: 1. does the
    kernel own secure state; 2. does the kernel own hyp mode.
    Where should this information be stored ?

 3. Is Linux allowed to assume that the secondary CPUs have the
    same properties as the boot CPU ?  If not, where do I store the
    availability of the secure/hyp modes for the secondary cpus ?
    Perhaps that ought to be in the device tree.

 4. I'm not very familiar with the KVM on ARM code.  How much would
    have to be changed to existing KVM on ARM to make it conform to
    the above scheme ?  In particular it would have to not use SMC to
    adjust the HVBAR; instead, it would have to take control of HVBAR
    once per CPU.

Opinions welcome.

Thanks,
Ian.

[1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment

ARM processor mode, kernel startup, Hyp / secure state

[Russell King - ARM Linux]

On Tue, Aug 23, 2011 at 03:52:25PM +0100, Ian Jackson wrote:
> We're looking into porting Xen to the ARM A15 architecture.  In this
> context, it's necessary to arrange for the Xen hypervisor to be
> entered from the boot loader in an appropriate processor mode.
> 
> KVM needs to deal with the same problem, of course.  And any future
> Linux kernel feature which uses the Secure State does too.

We already have kernels which boot in both secure and non-secure mode
CPUs.  There's very little difference between the two from the core
ARM support code - except that we have expectations that appropriate
hardware work-arounds will have already been initialized by the
secure monitor (as there is no way for the kernel to be able to call
out to the secure monitor in a platform independent way.)

That's unfortunately one of the necessary evils of having the secure
stuff, which doesn't have any kind of defined API, getting in the way.

So, basically, which side of the secure division the kernel runs on
depends highly on the platform itself - and as the secure monitor APIs
are highly platform specific, there's no getting away from that.

If the kernel does not contain the appropriate platform specific secure
monitor API then it can't run in non-secure mode (or it can but the
functionality will be severely limited - no power management, and
in some cases buggy L2 cache.)  So you wouldn't _want_ to run such a
kernel if you value your data.

As far as I know, the kernel runs non-secure on all real silicon out in
the wild.  It's only ARMs evaluation boards and models where the kernel
runs on the secure side.

Given all that, there's not much point to publishing whether a kernel is
built for secure side or not.  It's tied extremely closely to the platform
itself, which is where most of the secure API calls already live.

[Android-virt] ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Christoffer Dall writes ("Re: [Android-virt] ARM processor mode, kernel startup, Hyp / secure state"):
> There are a couple of things with the existing KVM ARM approach with
> the trivial boot wrapper which need to be fixed, though: firstly,
> there should be separate trampolines for hypervisor mode and for
> secure state.  That allows the two features to be used independently.
> Secondly, the trivial trampolines should be part of the kernel proper
> and their lifetime should not extend across the bootloader interface.
> 
> How are you imagining the first point? To have a fixed address and
> being able to issue an HVC instead of SMC from within the kernel if
> booted in non-secure?

I was thinking that if the kernel is booted in hyp mode (non-secure
obviously, then), the kernel would set HVBAR to point to a little
pretend hypervisor vector table whose only functionality was to be
able to set HVBAR.  Then later, when you load the kvm module (say),
this would be used (once per cpu) to set HVBAR to the actual vector
table.

This is the only way to make things work if secure state is in use for
something else (whether by the firmware, or by some other kernel
feature).

> As it is now, I thought the kernel could run in both secure and
> non-secure mode. If this is true, is it not a simple and useful
> feature that should be preserved?

This is a good point.  Yes, the kernel can run in both secure and
non-secure states.  However, if you want to use kvm, this is no longer
the case.  kvm wants to use hyp mode and hyp mode is only available in
the non-secure state.  And if you let the kernel get properly booted
in secure state, switching it to non-secure state while it is running
would be quite painful - you'd have to do a very complicated dance to
make sure the non-secure state was set up just right, etc.

>  1. The kernel will advertise via ELF notes what modes it may be
>    started in.  The possible modes will be:
>       (i)   secure monitor mode
>       (ii)  non-secure hypervisor mode
>       (iii) non-secure kernel mode
> 
> hmm, so you would have to choose _one_ of these at compile time and
> your image is bound to be run in the specific mode? How about when
> you're actually running in a VM, should it then be in mode (iii)?

No, I'm proposing that you'd choose one _or more_ of these at compile
time.  Most modern Linux kernels would support all three.  The Xen
hypervisor (which has to be booted via the same interface) would
support only (ii) and (iii).

> What are the benefits from the kernel being aware of its mode if we
> ignore initialization and handle that somehow else?

I'm not sure what you mean by "being aware of its mode".  If you mean
"being aware of the security state", then there is no direct benefit
just of that /awareness/.  However, a kernel which knows that it is
running in non-secure state but which has also set up a hook to allow
it to use secure state, could later (eg when a kernel module is
loaded) use the secure state for exciting functionality (eg, crypto
key storage, or the like).

If you mean "being aware of kernel mode vs. hyp mode", then the answer
is that the kernel cannot generally run in hyp mode, but if you want
to use kvm it needs access to hyp mode.  So the startup arrangements
need to arrange that the kvm kernel module can get back into hyp mode
when it wants to.

> so you're suggesting a method that causes the kernel never to be
> able to run isolated in secure mode?

You're right that that was the implication of my proposal.  However,
if that's something that people want to do then my proposal needs to
be modified.  But to understand that better, I'd like to understand
why you would want to run the whole kernel in secure state.  What
would you use the non-secure state for ?

If this is indeed a requirement, one option would be to simply say
that a kernel may be started in secure kernel state.  But there is a
difficulty: the kernel needs to be told somehow that it is in secure
kernel state, so that it knows that it can use the non-secure state.

>  2. This cpu startup process must happen very early - before paging
>    is enabled, in any case, so before RAM is really available.
>    However, it produces two bits of information: 1. does the
>    kernel own secure state; 2. does the kernel own hyp mode.
>    Where should this information be stored ?
> 
> so, per (1) I would think no. (2) if the kernel uses KVM, yes.

Um.  What I meant is: these two bits of information vary according to
how the kernel has been booted.  They don't have fixed answers.

>  4. I'm not very familiar with the KVM on ARM code.  How much would
>    have to be changed to existing KVM on ARM to make it conform to
>    the above scheme ?  In particular it would have to not use SMC to
>    adjust the HVBAR; instead, it would have to take control of HVBAR
>    once per CPU.
> 
> This is a tiny adjustment - we were planning to fit this with the community wishes anyhow.

Right.

Thanks,
Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> We already have kernels which boot in both secure and non-secure mode
> CPUs.  There's very little difference between the two from the core
> ARM support code - except that we have expectations that appropriate
> hardware work-arounds will have already been initialized by the
> secure monitor (as there is no way for the kernel to be able to call
> out to the secure monitor in a platform independent way.)

Right.

> So, basically, which side of the secure division the kernel runs on
> depends highly on the platform itself - and as the secure monitor APIs
> are highly platform specific, there's no getting away from that.

It seems to me that from the PoV of Linux there are two kind of
platform: ones where the kernel is expected to own the secure state,
and ones where it is expected to run only in non-secure state.

But just because Linux owns the secure state doesn't mean that it
should run in that state most of the time.  The question is really
whether Linux would like to use the secure/non-secure division to
provide a more secure facility which other parts of the system can
use, or to use it to provide a more constrained/contained environment
in which to run less-trusted code.

In the latter case Linux would want to run in secure state all of the
time.  In the former it would want to run mostly in non-secure state.
But it is difficult to choose between these later than boot time
because switching a running system from the secure to non-secure state
is so complex.

> If the kernel does not contain the appropriate platform specific secure
> monitor API then it can't run in non-secure mode (or it can but the
> functionality will be severely limited - no power management, and
> in some cases buggy L2 cache.)  So you wouldn't _want_ to run such a
> kernel if you value your data.

So is the secure state used for the same kind of things as SMM is on
Intel, then ?  Perhaps I haven't understood the usual setups well
enough.

> Given all that, there's not much point to publishing whether a kernel is
> built for secure side or not.  It's tied extremely closely to the platform
> itself, which is where most of the secure API calls already live.

The point of my proposed ELF flag isn't to advertise that a kernel is
capable of running in the secure state.  As you say, where this is
possible, we don't need to advertise it.

The point of my ELF flag is to advertise that the kernel would like to
_know_ that it has control of the secure state; the kernel is told
this by the fact that it enters in _monitor mode_ rather than kernel
mode.  Obviously you shouldn't enter an unconsenting kernel in monitor
mode.

If the kernel is to have control of the secure state it needs to know
this, and to have a way of making use of it.  It looked to me like the
easiest way of doing this was entering it in monitor mode.  (Since a
kernel entered in kernel mode can't tell easily whether it's in secure
or non-secure state.)

Likewise, if the kernel is to have control of the hyp mode, it needs
to know that and have a way to use it.  So entering the kernel in hyp
mode seemed to me the easiest way.

I think I know what the hyp mode is for and how people will use it
:-).  But I don't really know what the secure state is for.

Thanks,
Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Dave Martin]

On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> We're looking into porting Xen to the ARM A15 architecture. ?In this
> context, it's necessary to arrange for the Xen hypervisor to be
> entered from the boot loader in an appropriate processor mode.
>
> KVM needs to deal with the same problem, of course. ?And any future
> Linux kernel feature which uses the Secure State does too.
>
> We are currently working with ARM's "Fast Model" of the Cortex A15, a
> software emulator. ?We're using it in the mode where you feed it an
> ELF and it loads it into simulated RAM and starts executing at the
> ELF entrypoint. ?It does this in the CPU's defined startup mode, which
> is Kernel mode in the Secure state.
>
> It seems that this environment is what the nascent KVM-on-A15
> developers [1] are using too. ?There is modified version of the tiny
> boot wrapper (the normal version of which just emulates the proper
> calling API for the kernel); it sets up a trampoline security monitor.
>
> We need to define a correct calling convention for the kernel which is
> compatible with old systems, but which also allows the booted kernel
> (Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of
> all the available facilities. ?The correct approach does seem to be to
> have Linux set itself up a trivial trampoline which allows the kernel
> to later regain the elevated privilege.
>
> There are a couple of things with the existing KVM ARM approach with
> the trivial boot wrapper which need to be fixed, though: firstly,
> there should be separate trampolines for hypervisor mode and for
> secure state. ?That allows the two features to be used independently.
> Secondly, the trivial trampolines should be part of the kernel proper
> and their lifetime should not extend across the bootloader interface.
>
> At first I thought that the best thing to do would be to boot the
> kernel in any suitable mode, and have the kernel automatically detect
> the starting mode. ?I started writing code in linux's head.S to do
> this. ?However, detecting whether we are in secure state is very
> difficult: it involves deliberately risking an undefined instruction
> trap. ?The code for this was getting rather long and involved.

There may be a safe way to do this check -- for example, on ARM1176
and Cortex-A8 there is a CP14 debug status/control register that you can
read which includes a flag indicating which world you're in.  This isn't part
of the architecture though and may be different/not possible on some
CPUs.

All in all, it's better to engineer things so that the check doesn't need to
be done at all.

> Also, unconditionally starting a kernel in hypervisor mode seems
> rather unfriendly. ?At the moment we unconditionally start it in the
> secure state and indeed in the current setup it seems to run entirely
> in secure state. ?It seems to me that the kernel should mostly run in
> non-secure state.
>
> So I propose the following approach:
>
> ?1. The kernel will advertise via ELF notes what modes it may be
> ? ?started in. ?The possible modes will be:
> ? ? ? (i) ? secure monitor mode
> ? ? ? (ii) ?non-secure hypervisor mode
> ? ? ? (iii) non-secure kernel mode

Note that in real deployments, the kernel is not an ELF image and
therefore cannot have notes.

Currently, I think the kernel doesn't really have any metadata readable
by the bootloader at all.


I think the Secure World / Normal World distinction may be a red
herring.  The kernel currently just stays in whatever world it was
started in, and doesn't have support for being the bridge between
worlds.  If the kernel is started in the Normal World, this will be
because something else occupies the Secure World, and so Linux
won't have direct access to that anyway.  Conversely, if Linux is
started in the Secure World, it can always access the monitor state
if this is really desired -- the monitor is at the same level of privilege
as the other Secure World privileged modes.

I haven't seen a compelling reason why this would need to change;
do you have a particular scenario in mind?

> ?2. The bootloader will select the first mode from the three listed
> ? ?above which is supported by both the processor and the kernel to
> ? ?be loaded, and transition the processor to that mode. ?If this
> ? ?involves dropping out of secure or hypervisor mode, it will put
> ? ?those modes permanently beyond use.
>
> ?3. The kernel will examine CPSR to determine which of the three
> ? ?possibilities above has happened, and:
> ? (a) If started in monitor mode:
> ? ? ? * Grant access to everything to non-secure state
> ? ? ? * Set the non-secure copies of the various CP15 registers
> ? ? ? ? which don't have a sane value on cpu reset
> ? ? ? * Install a trivial monitor vector which unconditionally
> ? ? ? ? copies r0 to MVBAR and returns
> ? ? ? * Switch to non-secure Hyp mode (if available) and do (b),
> ? ? ? ? or non-secure Kernel mode (if Hyp mode not supported).
> ? (b) If started in Hyp mode:
> ? ? ? * Install a trivial hypervisor vector which unconditionally
> ? ? ? ? copies r0 to HVBAR and returns

Starting the kernel in Hyp mode may be reasonable in this scenario.

I think this is actually backwards-compatible, because in both an
uncompressed kernel and a zImage, pretty much the first thing that
happens is a switch to SVC mode at present.  So an older kernel
will just transparently end up in SVC mode as if nothing unusual had
happened.

The boot procotol would need to define the initial hypervisor state:
basically, a dormant identity configuration with all traps and fancy
features turned off.

With only minor changes, I think a kernel supporting this boot protocol
could run the zImage decompressor in Hyp mode, and then set up
a stub HVC handler when reaching the main kernel entry point.
It's certainly worth investigating.

There may be architectural reasons why you can't run the decompressor in
Hyp mode, but I'm not aware of any off the top of my head.

> ? (c) Rest of startup.
>
> Questions:
>
> ?1. What do people think ? ?If this seems plausible I will prepare
> ? ?an RFC patch for Linux and the boot-wrapper.git.
>
> ?2. This cpu startup process must happen very early - before paging
> ? ?is enabled, in any case, so before RAM is really available.
> ? ?However, it produces two bits of information: 1. does the
> ? ?kernel own secure state; 2. does the kernel own hyp mode.
> ? ?Where should this information be stored ?
>
> ?3. Is Linux allowed to assume that the secondary CPUs have the
> ? ?same properties as the boot CPU ? ?If not, where do I store the
> ? ?availability of the secure/hyp modes for the secondary cpus ?
> ? ?Perhaps that ought to be in the device tree.

I think yes, for now.  Having a different level of access on different CPUs
is either an unnecessary hindrance or a security hole (depending on
your point of view), and such a system would not be SMP.

Whether Linux makes use of those facilities on all CPUs is up to Linux though.

It's up to the Secure firmware (if any) and the bootloader to get the
secondary CPUs into the appropriate state.

> ?4. I'm not very familiar with the KVM on ARM code. ?How much would
> ? ?have to be changed to existing KVM on ARM to make it conform to
> ? ?the above scheme ? ?In particular it would have to not use SMC to
> ? ?adjust the HVBAR; instead, it would have to take control of HVBAR
> ? ?once per CPU.
>
> Opinions welcome.
>
> Thanks,
> Ian.
>
> [1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment
>
> _______________________________________________
> boot-architecture mailing list
> boot-architecture at lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/boot-architecture
>



-- 
Dave Martin <dave.martin@linaro.org>
Linaro Kernel Working Group

--
http://www.linaro.org/ -- Open source software for ARM SoCs

http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg
http://www.linaro.org/linaro-blog/

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> I think the Secure World / Normal World distinction may be a red
> herring.  The kernel currently just stays in whatever world it was
> started in, and doesn't have support for being the bridge between
> worlds.  If the kernel is started in the Normal World, this will be
> because something else occupies the Secure World, and so Linux
> won't have direct access to that anyway.  Conversely, if Linux is
> started in the Secure World, it can always access the monitor state
> if this is really desired -- the monitor is at the same level of privilege
> as the other Secure World privileged modes.

If Linux is booted in the secure world, it will have to switch to the
non-secure world at boot if it is to use kvm.

> I haven't seen a compelling reason why this would need to change;
> do you have a particular scenario in mind?

The non-secure world doesn't have hyp mode, which is needed by kvm.

So if kvm might be loaded later (and we don't really want to rule that
out at boot time unless we can help it) then the kernel needs either
  - to be started in the secure world, to know this, and switch to
    the non-secure world while installing appropriate trampoline(s)
  - to be started in hyp mode, and switch to kernel mode while
    installing an appropriate trampoline
  - something more horrible (eg the bootloader provides the
    trampoline; or switch the running system to the non-secure world
    when the desire for kvm becomes evident)

> Starting the kernel in Hyp mode may be reasonable in this scenario.
> 
> I think this is actually backwards-compatible, because in both an
> uncompressed kernel and a zImage, pretty much the first thing that
> happens is a switch to SVC mode at present.  So an older kernel
> will just transparently end up in SVC mode as if nothing unusual had
> happened.

Do all relevant existing kernels unconditionally switch to kernel
mode at the start ?

> The boot procotol would need to define the initial hypervisor state:
> basically, a dormant identity configuration with all traps and fancy
> features turned off.

Right.  Helpfully, that is (more or less) the processor reset state.

> With only minor changes, I think a kernel supporting this boot protocol
> could run the zImage decompressor in Hyp mode, and then set up
> a stub HVC handler when reaching the main kernel entry point.
> It's certainly worth investigating.

I haven't checked the decompressor but I don't see why it wouldn't
work in hyp mode.  I can try it easily enough.

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

I wrote:
> I haven't checked the decompressor but I don't see why it wouldn't
> work in hyp mode.  I can try it easily enough.

I tried booting linux in non-secure hyp mode [1] and it seemed to work
just fine.  The kernel decompressed and booted up to trying to mount
its root fs, just like it does if you start it in secure kernel mode.

Ian.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/linux-2.6-cm.git
    a104d064b95e4981619c66bb94840ab3cb67c75d

ARM processor mode, kernel startup, Hyp / secure state

[Catalin Marinas]

Ian,

(I'll be on holiday from tomorrow, so not able to follow up until September).

On 23 August 2011 15:52, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> We're looking into porting Xen to the ARM A15 architecture. ?In this
> context, it's necessary to arrange for the Xen hypervisor to be
> entered from the boot loader in an appropriate processor mode.

We had some discussions both with ARM partners and Christoffer and it
is not clear how we go about initialising the Hyp mode. Most SoC
vendors would probably just run Linux in non-secure SVC mode and would
not touch the Hyp mode at all. Linux would issue an SMC to set the
HVBAR (could do HVC but I expect that there is no secure monitor
code). Once this is set, assuming that the Hyp MMU is disabled, Linux
can invoke an HVC and initialise the Hyp mode.

The problem is that the SMC API is specific to a secure monitor
implementation. An alternative would be to ask the vendors to get some
simple Hyp code which would trap access to HVBAR for example and set
it with that value.

> So I propose the following approach:
>
> ?1. The kernel will advertise via ELF notes what modes it may be
> ? ?started in. ?The possible modes will be:
> ? ? ? (i) ? secure monitor mode
> ? ? ? (ii) ?non-secure hypervisor mode
> ? ? ? (iii) non-secure kernel mode

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> There may be a safe way to do this check -- for example, on ARM1176
> and Cortex-A8 there is a CP14 debug status/control register that you can
> read which includes a flag indicating which world you're in.  This isn't part
> of the architecture though and may be different/not possible on some
> CPUs.

Yes.  I don't think we can write this into a boot protocol spec unless
we have some architectural way of doing it.

> All in all, it's better to engineer things so that the check doesn't need to
> be done at all.

Yes, in principle.

> Note that in real deployments, the kernel is not an ELF image and
> therefore cannot have notes.

Yes.  But in the future it will do, surely ?  I mean, we're working on
bootloader protocols and this part (to do with startup cpu states and
ownership of the various privilege levels) is only part of it.

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> (I'll be on holiday from tomorrow, so not able to follow up until September).

Right.  Let's try to be quick :-).

> We had some discussions both with ARM partners and Christoffer and it
> is not clear how we go about initialising the Hyp mode. Most SoC
> vendors would probably just run Linux in non-secure SVC mode and would
> not touch the Hyp mode at all. Linux would issue an SMC to set the
> HVBAR (could do HVC but I expect that there is no secure monitor
> code). Once this is set, assuming that the Hyp MMU is disabled, Linux
> can invoke an HVC and initialise the Hyp mode.

Well, yes.  Do SoC vendors all expect to own the secure state ?

If so then the answer is perhaps simply that they should boot Linux in
Hyp mode (and expect Linux to drop to ordinary svc mode).

> The problem is that the SMC API is specific to a secure monitor
> implementation. An alternative would be to ask the vendors to get some
> simple Hyp code which would trap access to HVBAR for example and set
> it with that value.

That's seems to be extending the range of ways in which the firmware
can make things go wrong, and broadening the interface to the
firmware.

> From the above, I would just allow (iii). As Dave pointed out, we
> don't have the ELF information at this point.

If we had a more functional bootloader setup (which AIUI is definitely
planned) we could do.

> > ? (b) If started in Hyp mode:
> > ? ? ? * Install a trivial hypervisor vector which unconditionally
> > ? ? ? ? copies r0 to HVBAR and returns
> > ? (c) Rest of startup.
> 
> Maybe we could allow (b) if SoC vendors don't provide a different API
> to set the HVBAR. But it means that kernels not aware of this feature
> would fail.

It turns out that at least your kernel boots just fine if started in
Hyp mode :-).

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Russell King - ARM Linux]

On Tue, Aug 23, 2011 at 05:50:10PM +0100, Ian Jackson wrote:
> Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> > We already have kernels which boot in both secure and non-secure mode
> > CPUs.  There's very little difference between the two from the core
> > ARM support code - except that we have expectations that appropriate
> > hardware work-arounds will have already been initialized by the
> > secure monitor (as there is no way for the kernel to be able to call
> > out to the secure monitor in a platform independent way.)
> 
> Right.
> 
> > So, basically, which side of the secure division the kernel runs on
> > depends highly on the platform itself - and as the secure monitor APIs
> > are highly platform specific, there's no getting away from that.
> 
> It seems to me that from the PoV of Linux there are two kind of
> platform: ones where the kernel is expected to own the secure state,
> and ones where it is expected to run only in non-secure state.

The distinction is a highly technical one, and it comes from the lack
of any kind of secure monitor API, and the fact that the non-secure
world is prevented from accessing various registers.

A secure side kernel contains no calls to the secure monitor.  Calls
to the secure monitor are required to reprogram critical registers such
as L2 cache configuration registers and the like.  For example, on
OMAP, it is impossible to reliably invalidate the contents of the L2
cache without making secure mode calls to reprogram various control
registers.

Not having that register reprogrammed means you're running against
data corrupting errata bugs.

So, basically, a kernel built for non-secure will only run on the
non-secure side.  A kernel built for secure will only run reliably
on the secure side.

> But just because Linux owns the secure state doesn't mean that it
> should run in that state most of the time.  The question is really
> whether Linux would like to use the secure/non-secure division to
> provide a more secure facility which other parts of the system can
> use, or to use it to provide a more constrained/contained environment
> in which to run less-trusted code.

What you're talking about is using Linux as the secure monitor, and
effectively matching the dividing line between Linux and userspace
with the secure/non-secure division.

Remember that Linux is not a micro-kernel, so any other partitioning
doesn't make sense.  (To place the partition elsewhere implies that
you have to carve the kernel data structures up and contain them
within each world - with separate memory allocators etc.)

> > If the kernel does not contain the appropriate platform specific secure
> > monitor API then it can't run in non-secure mode (or it can but the
> > functionality will be severely limited - no power management, and
> > in some cases buggy L2 cache.)  So you wouldn't _want_ to run such a
> > kernel if you value your data.
> 
> So is the secure state used for the same kind of things as SMM is on
> Intel, then ?  Perhaps I haven't understood the usual setups well
> enough.

You're assuming I have deep knowledge of x86.  I don't, sorry.

> I think I know what the hyp mode is for and how people will use it
> :-).  But I don't really know what the secure state is for.

Secure state is used to run the secure monitor, which is a chunk of code
provided by the SoC manufacturer and normally programmed into ROM on real
SoCs.

What is does is up to the SoC manufacturer.  What services it provides is
up to the SoC manufacturer.

The implication is that once the secure monitor has set things up, it
will start execution of things like boot loaders, etc in non-secure
mode.  Moreover, there's usually no way to get back into secure mode
to run your own code (otherwise it wouldn't be secure...)

A SoC manufacturer may provide a secure monitor to deal with storing
secret keys, and providing some kind of encryption services.  Or it
may provide nothing more than a way to access system registers which
are only accessible from secure mode.

ARM processor mode, kernel startup, Hyp / secure state

[Russell King - ARM Linux]

On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
> Maybe we could allow (b) if SoC vendors don't provide a different API
> to set the HVBAR. But it means that kernels not aware of this feature
> would fail.

Oh, does this mean that ARM Ltd are starting to see the light in having
a common defined secure API at last - at least a subset - something which
I've mentioned several times...

ARM processor mode, kernel startup, Hyp / secure state

[Russell King - ARM Linux]

On Tue, Aug 23, 2011 at 06:18:37PM +0100, Ian Jackson wrote:
> Yes.  But in the future it will do, surely ?  I mean, we're working on
> bootloader protocols and this part (to do with startup cpu states and
> ownership of the various privilege levels) is only part of it.

No - at least not yet, and I don't see any reason why we'd ever want to
use raw ELF.

The problem with ELF is that it carries lots of baggage around which
platforms just don't need.  Fine if it's on a hard disk and you're
booting a desktop, but not for embedded where space tends to be a
premium.

And most of that additional data just isn't required.

Plus, I'd point out that _no one_ is talking about boot protocols.  Not
outside linaro at any rate.  Which means that they're doing effective
closed-door discussion.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> A secure side kernel contains no calls to the secure monitor.  Calls
> to the secure monitor are required to reprogram critical registers such
> as L2 cache configuration registers and the like.  For example, on
> OMAP, it is impossible to reliably invalidate the contents of the L2
> cache without making secure mode calls to reprogram various control
> registers.
> 
> Not having that register reprogrammed means you're running against
> data corrupting errata bugs.
> 
> So, basically, a kernel built for non-secure will only run on the
> non-secure side.  A kernel built for secure will only run reliably
> on the secure side.

It's not clear to me from reading the ARM ARM that a kernel written
for the non-secure side couldn't make the same monitor calls, but I
haven't seen any of the OMAP manuals and I haven't read any of the
errata (which you obviously have).

If there are kernels intended to run on the secure side, a difficulty
arises: if KVM wants to use Hyp mode and the associated features, how
is that to be done ?

> > But just because Linux owns the secure state doesn't mean that it
> > should run in that state most of the time.  The question is really
> > whether Linux would like to use the secure/non-secure division to
> > provide a more secure facility which other parts of the system can
> > use, or to use it to provide a more constrained/contained environment
> > in which to run less-trusted code.
> 
> What you're talking about is using Linux as the secure monitor, and
> effectively matching the dividing line between Linux and userspace
> with the secure/non-secure division.

Well, yes, that would be one way of looking at it.  I'm not sure why
you would want to do that, but it was just one of the two options if
Linux is trying to own the secure/non-secure distinction.

On the other hand if Linux wants to own that boundary and to use it to
provide a /more/ privileged facility, that would involve some Linux
kernel module hoisting a pile of its code (or, more likely, code
provided from userspace) into secure state, which code then pulled the
ladder up after itself.  The resulting thing wouldn't look much like a
user process and wouldn't (afterwards) be subject to unwanted
interference.  But I don't think anyone is doing anything like this ?

> > So is the secure state used for the same kind of things as SMM is on
> > Intel, then ?  Perhaps I haven't understood the usual setups well
> > enough.
> 
> You're assuming I have deep knowledge of x86.  I don't, sorry.

Fair enough.  (My own x86 knowledge is pretty shallow too TBH.)

> Secure state is used to run the secure monitor, which is a chunk of code
> provided by the SoC manufacturer and normally programmed into ROM on real
> SoCs.

If this is always (or generally) the case then Linux needs to run on
only one side of the secure/non-secure barrier.  Since Linux might
want to run kvm, which requires Hyp mode, which is only on the
non-secure side, that means it needs to run on the non-secure side.

What have I missed ?

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> On Tue, Aug 23, 2011 at 06:18:37PM +0100, Ian Jackson wrote:
> > Yes.  But in the future it will do, surely ?  I mean, we're working on
> > bootloader protocols and this part (to do with startup cpu states and
> > ownership of the various privilege levels) is only part of it.
> 
> No - at least not yet, and I don't see any reason why we'd ever want to
> use raw ELF.
> 
> The problem with ELF is that it carries lots of baggage around which
> platforms just don't need.  Fine if it's on a hard disk and you're
> booting a desktop, but not for embedded where space tends to be a
> premium.

Hrm, yes.

> Plus, I'd point out that _no one_ is talking about boot protocols.  Not
> outside linaro at any rate.  Which means that they're doing effective
> closed-door discussion.

I'm not sure whether by "outside linaro" you mean "outside some kind
of top secret Linaro meetings".  I went to an open meeting of Linaro
developers touching on exactly these kind of topics, and everything
was public and there was substantial interest.

I'm not a Linaro developer; but I have CC'd their boot-architecture
list which AIUI is where some of the Linaro folks were thinking of
discussing boot protocols.

Just to be clear: I don't have a particular axe to grind, except that
I would like there to be some kind of defined and reasonably stable
interface, and that obviously we (Xen.org) need that interface to
support access to Hyp mode (which is something that Linux needs too if
we are to have a properly supported KVM on ARM).

Are linux-arm-kernel at infradead and boot-architecture at linaro the right
places to try to have this conversation ?

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Dave Martin]

On Tue, Aug 23, 2011 at 5:59 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
>> I think the Secure World / Normal World distinction may be a red
>> herring. ?The kernel currently just stays in whatever world it was
>> started in, and doesn't have support for being the bridge between
>> worlds. ?If the kernel is started in the Normal World, this will be
>> because something else occupies the Secure World, and so Linux
>> won't have direct access to that anyway. ?Conversely, if Linux is
>> started in the Secure World, it can always access the monitor state
>> if this is really desired -- the monitor is at the same level of privilege
>> as the other Secure World privileged modes.
>
> If Linux is booted in the secure world, it will have to switch to the
> non-secure world at boot if it is to use kvm.
>
>> I haven't seen a compelling reason why this would need to change;
>> do you have a particular scenario in mind?
>
> The non-secure world doesn't have hyp mode, which is needed by kvm.

You mean "the secure world doesn't have hyp mode"?  Fair point if so;
I was forgetting this.

>
> So if kvm might be loaded later (and we don't really want to rule that
> out at boot time unless we can help it) then the kernel needs either
> ?- to be started in the secure world, to know this, and switch to
> ? ?the non-secure world while installing appropriate trampoline(s)
> ?- to be started in hyp mode, and switch to kernel mode while
> ? ?installing an appropriate trampoline

This feels like the preferable approach -- it makes no sense to require
the kernel to be started in the Secure World if we want to install
a hypervisor.

There might be some sense in *requiring* the bootloader to start the kernel
in the Normal World and in hyp mode if kvm (or other kernel-hosted hypervisor)
is to be used.  If you weren't booted this way then tough, you can't
use kvm.

If you allow the kernel to be started in the Secure World, it can switch
to the Normal World on boot, but this is in practice a complex,
platform-specific procecure: you need to reprogram protection controllers
etc. to move all the peripherals and RAM into the Normal World,

Currently, I believe none of the needed code to accomplish this is in
the kernel, because there's been no prior need for it.  Implemting this
would potentially require work for every board >= ARMv6Z.
Bootloader maintainers might argue with that though: general-purpose
bootloaders probably won't have this code either yet, for the same reasons.
If there is real secure firmware, it will of course set all this up before
invoking the bootloader.  But otherwise, _someone_ has to do it...

> ?- something more horrible (eg the bootloader provides the
> ? ?trampoline; or switch the running system to the non-secure world
> ? ?when the desire for kvm becomes evident)

I agree with "horrible" :)

In particular, switching to the Normal World after boot is going to be
extremely horrible given the need to move peripherals and RAM into
the Normal world while somehow not pulling the floor from under your
feet...

>
>> Starting the kernel in Hyp mode may be reasonable in this scenario.
>>
>> I think this is actually backwards-compatible, because in both an
>> uncompressed kernel and a zImage, pretty much the first thing that
>> happens is a switch to SVC mode at present. ?So an older kernel
>> will just transparently end up in SVC mode as if nothing unusual had
>> happened.
>
> Do all relevant existing kernels unconditionally switch to kernel
> mode at the start ?

In practice, I think so.  Take a look at arch/arm/kernel/head.S and
arch/arm/boot/compressed/head.S.  Really, it's interrupts which get
disabled at these entry points -- but prior to ARMv6 it used to be
simpler to set the mode too since you couldn't force the interrupt
mask bits using a single MSR instruction without also setting the
processor mode.

Looking at compressed/head.S, it looks like the CPSR is only touched
by read-modify-write on the interrupt bits, or not at all.  So this
code is probably transparent to the initial CPU mode without
modification.

>
>> The boot procotol would need to define the initial hypervisor state:
>> basically, a dormant identity configuration with all traps and fancy
>> features turned off.
>
> Right. ?Helpfully, that is (more or less) the processor reset state.

Even if it isn't, I think it makes sense to place this requirement on
the bootloader.
But you're probably right.

Cheers
---Dave

ARM processor mode, kernel startup, Hyp / secure state

[Nicolas Pitre]

On Tue, 23 Aug 2011, Ian Jackson wrote:

> If there are kernels intended to run on the secure side, a difficulty
> arises: if KVM wants to use Hyp mode and the associated features, how
> is that to be done ?

I think that you can safely presume that the secure world is not open to 
Open Source in practice.  As Russell said, all the actual 
implementations meant for products out there have the secure mode 
effectively burried behind ROM code that the SOC vendor controls.


Nicolas

ARM processor mode, kernel startup, Hyp / secure state

[Nicolas Pitre]

On Tue, 23 Aug 2011, Ian Jackson wrote:
> Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> > Plus, I'd point out that _no one_ is talking about boot protocols.  Not
> > outside linaro at any rate.  Which means that they're doing effective
> > closed-door discussion.
> 
> I'm not sure whether by "outside linaro" you mean "outside some kind
> of top secret Linaro meetings".  I went to an open meeting of Linaro
> developers touching on exactly these kind of topics, and everything
> was public and there was substantial interest.

For the record, this Linaro topic and the progress done is documented here:
https://blueprints.launchpad.net/linaro/+spec/linaro-kernel-o-bootarchitecture

Nothing (yet) in there that would highly passionate you Russell I'm 
afraid.  This is probably why you didn't hear of it before.  But the 
discussion and the mailing list is public and open to anyone interested.

> I'm not a Linaro developer; but I have CC'd their boot-architecture
> list which AIUI is where some of the Linaro folks were thinking of
> discussing boot protocols.
> 
> Just to be clear: I don't have a particular axe to grind, except that
> I would like there to be some kind of defined and reasonably stable
> interface, and that obviously we (Xen.org) need that interface to
> support access to Hyp mode (which is something that Linux needs too if
> we are to have a properly supported KVM on ARM).
> 
> Are linux-arm-kernel at infradead and boot-architecture at linaro the right
> places to try to have this conversation ?

Absolutely.


Nicolas

ARM processor mode, kernel startup, Hyp / secure state

[Catalin Marinas]

On 23 August 2011 18:23, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
>> We had some discussions both with ARM partners and Christoffer and it
>> is not clear how we go about initialising the Hyp mode. Most SoC
>> vendors would probably just run Linux in non-secure SVC mode and would
>> not touch the Hyp mode at all. Linux would issue an SMC to set the
>> HVBAR (could do HVC but I expect that there is no secure monitor
>> code). Once this is set, assuming that the Hyp MMU is disabled, Linux
>> can invoke an HVC and initialise the Hyp mode.
>
> Well, yes. ?Do SoC vendors all expect to own the secure state ?

Maybe some parts of it, that's why it's hard to standardise some kind
of API. Since the platform boots in secure mode, they need at least
some code which they own, in addition to other stuff like secure
transactions.

> If so then the answer is perhaps simply that they should boot Linux in
> Hyp mode (and expect Linux to drop to ordinary svc mode).

That would be a solution as it is relatively simple for Linux to check
the mode and switch to SVC. The issue I see is that Linux is not the
first non-secure code to run but you usually get some boot loader like
U-Boot. You would have to make the latter aware of the Hyp mode as
well and many SoC vendors may not be willing to put the effort in (we
can help by providing boot loader fixes but the boot loader may not
always be open source).

(I may get back online towards the weekend)

-- 
Catalin

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

Dave,

On Tue, Aug 23, 2011 at 05:50:19PM +0100, Dave Martin wrote:
> On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > At first I thought that the best thing to do would be to boot the
> > kernel in any suitable mode, and have the kernel automatically detect
> > the starting mode. ?I started writing code in linux's head.S to do
> > this. ?However, detecting whether we are in secure state is very
> > difficult: it involves deliberately risking an undefined instruction
> > trap. ?The code for this was getting rather long and involved.
> 
> There may be a safe way to do this check -- for example, on ARM1176
> and Cortex-A8 there is a CP14 debug status/control register that you can
> read which includes a flag indicating which world you're in.  This isn't part
> of the architecture though and may be different/not possible on some
> CPUs.

Please don't do this! Accessing the debug registers via the CP14 registers
is like playing russian roulette with a machine gun, especially when you
have various hypervisor registers and hardware lock registers to contend
with. For 3.2, I will be guarding all of the hw_breakpoint init debug
probing with an undef_hook because I'm sick of blowing my head off when
systems are configured to keep debug out.

> All in all, it's better to engineer things so that the check doesn't need to
> be done at all

Agreed. I think it's better to assume that you can't detect whether you're
running in secure state or not.

Will

ARM processor mode, kernel startup, Hyp / secure state

[Dave Martin]

On Tue, Aug 23, 2011 at 11:18:00PM +0100, Will Deacon wrote:
> Dave,
> 
> On Tue, Aug 23, 2011 at 05:50:19PM +0100, Dave Martin wrote:
> > On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > > At first I thought that the best thing to do would be to boot the
> > > kernel in any suitable mode, and have the kernel automatically detect
> > > the starting mode. ?I started writing code in linux's head.S to do
> > > this. ?However, detecting whether we are in secure state is very
> > > difficult: it involves deliberately risking an undefined instruction
> > > trap. ?The code for this was getting rather long and involved.
> > 
> > There may be a safe way to do this check -- for example, on ARM1176
> > and Cortex-A8 there is a CP14 debug status/control register that you can
> > read which includes a flag indicating which world you're in.  This isn't part
> > of the architecture though and may be different/not possible on some
> > CPUs.
> 
> Please don't do this! Accessing the debug registers via the CP14 registers
> is like playing russian roulette with a machine gun, especially when you
> have various hypervisor registers and hardware lock registers to contend
> with. For 3.2, I will be guarding all of the hw_breakpoint init debug
> probing with an undef_hook because I'm sick of blowing my head off when
> systems are configured to keep debug out.

Heh -- I didn't think it was quite that scary, but point taken.

> > All in all, it's better to engineer things so that the check doesn't need to
> > be done at all
> 
> Agreed. I think it's better to assume that you can't detect whether you're
> running in secure state or not.
> 
> Will

ARM processor mode, kernel startup, Hyp / secure state

[Linus Walleij]

On Tue, Aug 23, 2011 at 7:42 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
>> Maybe we could allow (b) if SoC vendors don't provide a different API
>> to set the HVBAR. But it means that kernels not aware of this feature
>> would fail.
>
> Oh, does this mean that ARM Ltd are starting to see the light in having
> a common defined secure API at last - at least a subset - something which
> I've mentioned several times...

I've discussed this internally at ST-Ericsson at times.

There is something called the Trusted Execution Environment
(TEE) which is standardized by Global Platform, a consortia
where ARM are members, as are ST, Ericsson, NXP, Renesas,
and TI (etc):
http://www.globalplatform.org/membershipcurrentfull.asp

For Ux500 we have a driver for this API which can be found in
an out-of-tree git, basically for Ux500 the ROM implements
the service tunnel, and we call into the ROM to perform them.
The ROM in turn will issue an SMC to execute the actual
call to the secure world.

The TEE defines Trusted Applications (TA:s) that can be
called, they use a GUID identifier on each side as handle.
I think of it as some exec() call which returns the result in
a pipe ... (I don't know if this analogy is correct)

Right now these TEE calls are mostly for performing this or
that cryptographic operation on some buffer of data, keeping
the secret crypto keys in the secure world. These calls are
done from userspace with the kernel driver simply
mediating them.

The basic question to which I have no answer is whether
TEE is or is not intended to be used for low-level services
like l2x0 on/off or so. Or if the idea never even crossed the
minds of the people devising TEE, like say if they were
having totally different use cases in their mind when they
cooked it up.

If it was intended for such stuff, we/GlobalPlatform could
define a few standard "TAs" that all platforms would
implement, unless it's by nature too heavyweight for that.

Would be fun if someone with insight could fill in on
this...

[TEE experts, smack my fingers if I got some detail
wrong in here.]

Yours,
Linus Walleij

ARM processor mode, kernel startup, Hyp / secure state

[Martin HOVANG]

Hi,

The GlobalPlatform TEE spec do not address this kinds of functions. The Trusted Applications (TAs) defined by this initiative runs in user space and do not have the rights to do poke in hw. An extension of the GP framework to support applications running in the TEE Core (supervisor mode) would be needed. This extension could be what we have running on the 8500 today. Basically we use the same GP Client API (Linux side) with the addition to the secure side OS to access functions running in supervisor mode. I.e. the secure side implementation is extended without creating a conflict or changing the GP Client API in Linux.

Regards
/Martin

-----Original Message-----
From: Linus Walleij [mailto:linus.walleij at linaro.org] 
Sent: den 24 augusti 2011 13:43
To: Russell King - ARM Linux
Cc: Catalin Marinas; boot-architecture at lists.linaro.org; Ian Jackson; linux-arm-kernel at lists.infradead.org; android-virt at lists.cs.columbia.edu; Hakan ENGLUND; Martin HOVANG
Subject: Re: ARM processor mode, kernel startup, Hyp / secure state

On Tue, Aug 23, 2011 at 7:42 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
>> Maybe we could allow (b) if SoC vendors don't provide a different API
>> to set the HVBAR. But it means that kernels not aware of this feature
>> would fail.
>
> Oh, does this mean that ARM Ltd are starting to see the light in having
> a common defined secure API at last - at least a subset - something which
> I've mentioned several times...

I've discussed this internally at ST-Ericsson at times.

There is something called the Trusted Execution Environment
(TEE) which is standardized by Global Platform, a consortia
where ARM are members, as are ST, Ericsson, NXP, Renesas,
and TI (etc):
http://www.globalplatform.org/membershipcurrentfull.asp

For Ux500 we have a driver for this API which can be found in
an out-of-tree git, basically for Ux500 the ROM implements
the service tunnel, and we call into the ROM to perform them.
The ROM in turn will issue an SMC to execute the actual
call to the secure world.

The TEE defines Trusted Applications (TA:s) that can be
called, they use a GUID identifier on each side as handle.
I think of it as some exec() call which returns the result in
a pipe ... (I don't know if this analogy is correct)

Right now these TEE calls are mostly for performing this or
that cryptographic operation on some buffer of data, keeping
the secret crypto keys in the secure world. These calls are
done from userspace with the kernel driver simply
mediating them.

The basic question to which I have no answer is whether
TEE is or is not intended to be used for low-level services
like l2x0 on/off or so. Or if the idea never even crossed the
minds of the people devising TEE, like say if they were
having totally different use cases in their mind when they
cooked it up.

If it was intended for such stuff, we/GlobalPlatform could
define a few standard "TAs" that all platforms would
implement, unless it's by nature too heavyweight for that.

Would be fun if someone with insight could fill in on
this...

[TEE experts, smack my fingers if I got some detail
wrong in here.]

Yours,
Linus Walleij

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

Hi Ian,

On Tue, Aug 23, 2011 at 06:23:32PM +0100, Ian Jackson wrote:
> Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> > (I'll be on holiday from tomorrow, so not able to follow up until September).
> 
> Right.  Let's try to be quick :-).

Ha, you've got enough people interested now that there's no need to rush!

> > We had some discussions both with ARM partners and Christoffer and it
> > is not clear how we go about initialising the Hyp mode. Most SoC
> > vendors would probably just run Linux in non-secure SVC mode and would
> > not touch the Hyp mode at all. Linux would issue an SMC to set the
> > HVBAR (could do HVC but I expect that there is no secure monitor
> > code). Once this is set, assuming that the Hyp MMU is disabled, Linux
> > can invoke an HVC and initialise the Hyp mode.
> 
> Well, yes.  Do SoC vendors all expect to own the secure state ?
> 
> If so then the answer is perhaps simply that they should boot Linux in
> Hyp mode (and expect Linux to drop to ordinary svc mode).

I think it's important to separate the problems of secure boot with the
problems of installing a hypervisor. Whatever happens in secure world, we can
expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev
board you might run directly in the secure world so there's a bit of extra
work to do to get out of that but then we can just drop into HVC mode and
forget about it.

> If we had a more functional bootloader setup (which AIUI is definitely
> planned) we could do.
> 
> > > ? (b) If started in Hyp mode:
> > > ? ? ? * Install a trivial hypervisor vector which unconditionally
> > > ? ? ? ? copies r0 to HVBAR and returns
> > > ? (c) Rest of startup.

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> From discussions that I've been having with the technical architects at ARM
> and various bootloader people, it looks like we have two scenarios [this is
> reiterating a lot of what you've said but I think it's important]:

Right.

>   After the system has taken care of whatever secure initialisation was
>   required, it moves into HYP mode. At this point there are two things that
>   can happen:
> 
>   1.0: HYP mode boot loader (uboot, UEFI) runs and installs Linux at the
>        same privilege level
>   1.1: Raw Linux boots, and detects HYP mode
>   1.2: Sets up basic init, with HVC trampoline for installing KVM/Xen later
>   1.3: Switch to SVC mode
>   1.4: Continue booting Linux as normal

Since we have discovered that this approach will work fine with
existing kernels, even ones which are completely unaware of Hyp mode,
I think this is the right approach.  It can be implemented by boot
loaders etc. right away.

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

On Wed, Aug 24, 2011 at 04:45:48PM +0100, Ian Jackson wrote:
> Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> >   After the system has taken care of whatever secure initialisation was
> >   required, it moves into HYP mode. At this point there are two things that
> >   can happen:
> > 
> >   1.0: HYP mode boot loader (uboot, UEFI) runs and installs Linux at the
> >        same privilege level
> >   1.1: Raw Linux boots, and detects HYP mode
> >   1.2: Sets up basic init, with HVC trampoline for installing KVM/Xen later
> >   1.3: Switch to SVC mode
> >   1.4: Continue booting Linux as normal
> 
> Since we have discovered that this approach will work fine with
> existing kernels, even ones which are completely unaware of Hyp mode,
> I think this is the right approach.  It can be implemented by boot
> loaders etc. right away.

It's certainly the simplest method from our point of view, but there is
still the necessity to support the other situation I described rather than
retrofit it when hardware that does it turns up [which it will].

The next step is to define our interface to the HVC trampoline. I think it
would be good if this interface could be reused by vendors who provide their
own hypervisor layer as this would reduce the need for lots of distinct HVC
APIs that essentially all do the same thing. It might be a good idea to
`reserve' a high region of the HVC space for use by Linux rather than index
the calls from 0, for example, which is likely to conflict with other
hypervisor implementations. I think the trampoline only needs a handful of
HVC calls (if that) anyway.

What do you reckon?

Will

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> The next step is to define our interface to the HVC trampoline. I think it
> would be good if this interface could be reused by vendors who provide their
> own hypervisor layer as this would reduce the need for lots of distinct HVC
> APIs that essentially all do the same thing. It might be a good idea to
> `reserve' a high region of the HVC space for use by Linux rather than index
> the calls from 0, for example, which is likely to conflict with other
> hypervisor implementations. I think the trampoline only needs a handful of
> HVC calls (if that) anyway.
> 
> What do you reckon?

I think the trampoline needs to have one hypercall: "set HVBAR".
(And, implicitly, it needs to be specified that the resulting vectors
will run without physical-to-virtual page translation, etc.)  That is
sufficient to bootstrap all the rest of the way up - and that other
bootstrapping code would have to exist anyway.

So I'm imagining that the kvm kernel module would, when loaded,
make that "set HVBAR" once on each cpu, replacing the trampoline
hypervisor with its own set of hypercall vectors.

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
> Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> > The next step is to define our interface to the HVC trampoline. I think it
> > would be good if this interface could be reused by vendors who provide their
> > own hypervisor layer as this would reduce the need for lots of distinct HVC
> > APIs that essentially all do the same thing. It might be a good idea to
> > `reserve' a high region of the HVC space for use by Linux rather than index
> > the calls from 0, for example, which is likely to conflict with other
> > hypervisor implementations. I think the trampoline only needs a handful of
> > HVC calls (if that) anyway.
> > 
> > What do you reckon?
> 
> I think the trampoline needs to have one hypercall: "set HVBAR".
> (And, implicitly, it needs to be specified that the resulting vectors
> will run without physical-to-virtual page translation, etc.)  That is
> sufficient to bootstrap all the rest of the way up - and that other
> bootstrapping code would have to exist anyway.

I also thought you might want a hypercall that gives you a version number
(i.e. a way to identify the hypervisor trampoline in case it changes) and we
might also want something for doing a system reset (for example, if a kernel
that hasn't installed a hypervisor wants to kexec another kernel image).

> So I'm imagining that the kvm kernel module would, when loaded,
> make that "set HVBAR" once on each cpu, replacing the trampoline
> hypervisor with its own set of hypercall vectors.

That's the plan!

Will

ARM processor mode, kernel startup, Hyp / secure state

[Russell King - ARM Linux]

On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
> I think it's important to separate the problems of secure boot with the
> problems of installing a hypervisor. Whatever happens in secure world, we can
> expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev
> board you might run directly in the secure world so there's a bit of extra
> work to do to get out of that but then we can just drop into HVC mode and
> forget about it.

I think you're painting a very simple picture there.

If the kernel has been handed over to while in secure mode, that's
probably because there is no secure monitor implemented.  If there's
no secure monitor implemented, there's no code in place to handle
SMC instructions.  To make things worse, if we drop out into non-secure
mode, due to the lack of working SMC instructions, we have no way to
access the various registers we need to.

So, if the kernel is booted in secure mode and wants to drop into non-
secure mode, we need a separate blob of code to install as our own
secure monitor to provide these services.

> Unlike the fragmented secure monitor API that currently exists across
> different platforms, it's really in the interests of the vendor to
> standardise on the HYP interface and provide calls to install code, otherwise
> they run the risk of producing what is essentially a closed system.

On the other hand, I'm sure vendors are already thinking along those
lines - the precident has been set by the secure monitor API mess, so
if it can be made to "work" there...

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

On Wed, Aug 24, 2011 at 06:09:53PM +0100, Russell King - ARM Linux wrote:
> On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
> > I think it's important to separate the problems of secure boot with the
> > problems of installing a hypervisor. Whatever happens in secure world, we can
> > expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev
> > board you might run directly in the secure world so there's a bit of extra
> > work to do to get out of that but then we can just drop into HVC mode and
> > forget about it.
> 
> I think you're painting a very simple picture there.

Yes, I was trying to avoid derailing the discussion because the secure -
non-secure transition should be treated separately to the hypervisor stuff.

> If the kernel has been handed over to while in secure mode, that's
> probably because there is no secure monitor implemented.  If there's
> no secure monitor implemented, there's no code in place to handle
> SMC instructions.  To make things worse, if we drop out into non-secure
> mode, due to the lack of working SMC instructions, we have no way to
> access the various registers we need to.
> 
> So, if the kernel is booted in secure mode and wants to drop into non-
> secure mode, we need a separate blob of code to install as our own
> secure monitor to provide these services.

Indeed. I'm not aware of any open-source monitor implementations and I'm not
sure whether it's something worth investing effort in.

> > Unlike the fragmented secure monitor API that currently exists across
> > different platforms, it's really in the interests of the vendor to
> > standardise on the HYP interface and provide calls to install code, otherwise
> > they run the risk of producing what is essentially a closed system.
> 
> On the other hand, I'm sure vendors are already thinking along those
> lines - the precident has been set by the secure monitor API mess, so
> if it can be made to "work" there...

That's partly why I'm trying to keep the two problems separate (but I doubt
it will make any difference!). That said, I firmly believe that whatever we
do, we *will* see platforms which install a hypervisor before dropping to
NS-SVC and booting Linux. We can either give up in these cases (which is
fine until the hardware becomes available and people want to use it) or we
can define a simple HVC interface that the kernel can use and hope people
implement that. If they don't, we'll end up in the same situation where
we are for the secure stuff but at least we tried. Unlike the SVC mess, you
can get away with a single (custom) HVC to bootstrap the thing with an
interface that we understand.

Will

ARM processor mode, kernel startup, Hyp / secure state

[Dave Martin]

On Wed, Aug 24, 2011 at 6:09 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
>> I think it's important to separate the problems of secure boot with the
>> problems of installing a hypervisor. Whatever happens in secure world, we can
>> expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev
>> board you might run directly in the secure world so there's a bit of extra
>> work to do to get out of that but then we can just drop into HVC mode and
>> forget about it.
>
> I think you're painting a very simple picture there.
>
> If the kernel has been handed over to while in secure mode, that's
> probably because there is no secure monitor implemented. ?If there's
> no secure monitor implemented, there's no code in place to handle
> SMC instructions. ?To make things worse, if we drop out into non-secure
> mode, due to the lack of working SMC instructions, we have no way to
> access the various registers we need to.
>
> So, if the kernel is booted in secure mode and wants to drop into non-
> secure mode, we need a separate blob of code to install as our own
> secure monitor to provide these services.
>
>> Unlike the fragmented secure monitor API that currently exists across
>> different platforms, it's really in the interests of the vendor to
>> standardise on the HYP interface and provide calls to install code, otherwise
>> they run the risk of producing what is essentially a closed system.
>
> On the other hand, I'm sure vendors are already thinking along those
> lines - the precident has been set by the secure monitor API mess, so
> if it can be made to "work" there...

A few thoughts from my end...

The kernel potentially has a lot of work to do if it is to support
virtualisation
when booted in the Secure World, because the kernel needs to switch to
the Normal World -- security controllers on the bus will need to be reprogrammed
etc., and the kernel would also need to retain some Secure memory somewhere
for handling SMC -- i.e., per-SoC code and configuration which just
doesn't exist
in the kernel yet.  The kernel does not need a "real" monitor in this
scenario, but
it will still need to get back into the Secure World to do certain
low-level operations
as we observe on e.g. OMAP.  This is not trivial since SMC causes a change of
memory space as well as privilege.

This potentially gets quite complex and messy, and the kernel doesn't really
care; my gut feeling is that we probably don't want to go there --
it's a fair amount
of extra code from which we don't really get much benefit.

So, it could be reasonable to _require_ linux to be started in the Normal
World if linux will install its own hypervisor, with suitable (possibly minimal)
SMC and HVC handlers/stubs provided by the bootloader or (more probably)
firmware.

Hopefully, those SMC/HVC interfaces will get reasonably standardised,
but in the meantime a bit of SoC-specific code in linux should be adequate
to smooth any differences.  Vendors likely have a strong incentive to provide
adequate and working functionality at these interfaces because failure to do
so could be a significant market disadvantage.

Cheers
---Dave

ARM processor mode, kernel startup, Hyp / secure state

[Ian Jackson]

Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
> > I think the trampoline needs to have one hypercall: "set HVBAR".
> > (And, implicitly, it needs to be specified that the resulting vectors
> > will run without physical-to-virtual page translation, etc.)  That is
> > sufficient to bootstrap all the rest of the way up - and that other
> > bootstrapping code would have to exist anyway.
> 
> I also thought you might want a hypercall that gives you a version number
> (i.e. a way to identify the hypervisor trampoline in case it changes)

Since the hypervisor trampoline was installed by the very same kernel
which is running, there is no need for any defined interface or
version numbers.

> and we might also want something for doing a system reset (for
> example, if a kernel that hasn't installed a hypervisor wants to
> kexec another kernel image).

No, if a kernel has installed a hypervisor and wishes to kexec then it
must start the new kernel in Hyp mode - since that is what we are
proposing to specify as the cpu mode to use when booting the kernel.

So the outgoing kernel simply switches to Hyp mode, tears down Hyp
mode's paging (and probably the rest of the paging too), and then
enters the new kernel in Hyp mode with paging off.  The new kernel
then installs its own hypervisor trampoline.

If the outgoing kernel hasn't installed a hypervisor it's even easier:
simply do the usual teardown, and then switch to Hyp mode using its
existing trampoline and then enter the new kernel.

Ian.

ARM processor mode, kernel startup, Hyp / secure state

[Will Deacon]

On Thu, Aug 25, 2011 at 11:13:40AM +0100, Ian Jackson wrote:
> Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
> > On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
> > > I think the trampoline needs to have one hypercall: "set HVBAR".
> > > (And, implicitly, it needs to be specified that the resulting vectors
> > > will run without physical-to-virtual page translation, etc.)  That is
> > > sufficient to bootstrap all the rest of the way up - and that other
> > > bootstrapping code would have to exist anyway.
> > 
> > I also thought you might want a hypercall that gives you a version number
> > (i.e. a way to identify the hypervisor trampoline in case it changes)
> 
> Since the hypervisor trampoline was installed by the very same kernel
> which is running, there is no need for any defined interface or
> version numbers.

Sure, but when we are confronted with a hypervisor that we didn't write
ourselves it might be nice if it looks the same as our Linux trampoline. If
we don't define this interface, that can never happen. So at the very least,
let's define the HVC number that we want to use for our single hypercall.

> > and we might also want something for doing a system reset (for
> > example, if a kernel that hasn't installed a hypervisor wants to
> > kexec another kernel image).
> 
> No, if a kernel has installed a hypervisor and wishes to kexec then it
> must start the new kernel in Hyp mode - since that is what we are
> proposing to specify as the cpu mode to use when booting the kernel.

Agreed.

> So the outgoing kernel simply switches to Hyp mode, tears down Hyp
> mode's paging (and probably the rest of the paging too), and then
> enters the new kernel in Hyp mode with paging off.  The new kernel
> then installs its own hypervisor trampoline.
> 
> If the outgoing kernel hasn't installed a hypervisor it's even easier:
> simply do the usual teardown, and then switch to Hyp mode using its
> existing trampoline and then enter the new kernel.

Thanks for the explanation. I was concerned about losing state across the
HVC but if it just returns back to you in the higher privilege level then
that's not a problem.

Will