Linux kernel set the hypervisor vector table through ATF

Linux kernel set the hypervisor vector table through ATF

[Dongjiu Geng]

Hi All,
      when Linux kernel boot from EL1,  there is no method to let
kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
interface between kernel and EL3 ATF to let kernel can set the
hypervisor vector table,  then can enter EL2 to enable hypervisor, as
shown in [1].
Do you agree?  Otherwise there is no method to enter EL2 hypervisor
when kernel boot from EL1, because the hypervisor vector
table(vbar_el2) is unknown.



[1]:
el1_sync:
        cmp     x0, #SMC_SET_VECTORS
        b.ne    2f
        msr     vbar_el2, x1
        b       9f

2:      cmp     x0, #SMC_SOFT_RESTART
        b.ne    3f
        mov     x0, x2
        mov     x2, x4
        mov     x4, x1
        mov     x1, x3
        br      x4                              // no return

3:      cmp     x0, #SMC_RESET_VECTORS
        beq     9f                              // Nothing to reset!

        ldr     x0, =SMC_STUB_ERR
        eret

9:      mov     x0, xzr
        eret
ENDPROC(el1_sync)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Mark Rutland]

On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> Hi All,
>       when Linux kernel boot from EL1,  there is no method to let
> kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> interface between kernel and EL3 ATF to let kernel can set the
> hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> shown in [1].
> Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> when kernel boot from EL1, because the hypervisor vector
> table(vbar_el2) is unknown.

The kernel already supported being booted at EL2, where it will install
itself as the hypervisor (and will drop to EL1 if required). EL2 is the
preferred boot mode, as we document in:

https://www.kernel.org/doc/html/latest/arm64/booting.html

... where we say:

| The CPU must be in either EL2 (RECOMMENDED in order to have access to
| the virtualisation extensions) or non-secure EL1.

We *strongly* prefer this over adding new ABIs to transition from EL1 to
EL2. Please boot the kernel at EL2 if you want to use KVM.

Thanks,
Mark.

> 
> 
> 
> [1]:
> el1_sync:
>         cmp     x0, #SMC_SET_VECTORS
>         b.ne    2f
>         msr     vbar_el2, x1
>         b       9f
> 
> 2:      cmp     x0, #SMC_SOFT_RESTART
>         b.ne    3f
>         mov     x0, x2
>         mov     x2, x4
>         mov     x4, x1
>         mov     x1, x3
>         br      x4                              // no return
> 
> 3:      cmp     x0, #SMC_RESET_VECTORS
>         beq     9f                              // Nothing to reset!
> 
>         ldr     x0, =SMC_STUB_ERR
>         eret
> 
> 9:      mov     x0, xzr
>         eret
> ENDPROC(el1_sync)
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Dongjiu Geng]

Mark Rutland <mark.rutland@arm.com> 于2021年6月1日周二 下午5:19写道：
>
> On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> > Hi All,
> >       when Linux kernel boot from EL1,  there is no method to let
> > kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> > interface between kernel and EL3 ATF to let kernel can set the
> > hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> > shown in [1].
> > Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> > when kernel boot from EL1, because the hypervisor vector
> > table(vbar_el2) is unknown.
>
> The kernel already supported being booted at EL2, where it will install
> itself as the hypervisor (and will drop to EL1 if required). EL2 is the
> preferred boot mode, as we document in:
>
> https://www.kernel.org/doc/html/latest/arm64/booting.html
>
> ... where we say:
>
> | The CPU must be in either EL2 (RECOMMENDED in order to have access to
> | the virtualisation extensions) or non-secure EL1.
>
> We *strongly* prefer this over adding new ABIs to transition from EL1 to
> EL2. Please boot the kernel at EL2 if you want to use KVM.

Thanks for the answer.
If use KVM, it should boot from EL2.  But if the hypervisor is not
KVM, such as Jailhouse hypervisor and some Chip manufacturer boot the
host kernel from EL1(not follow above rule), it seems there is not way
to enter the Jailhouse hypervisor.

>
> Thanks,
> Mark.
>
> >
> >
> >
> > [1]:
> > el1_sync:
> >         cmp     x0, #SMC_SET_VECTORS
> >         b.ne    2f
> >         msr     vbar_el2, x1
> >         b       9f
> >
> > 2:      cmp     x0, #SMC_SOFT_RESTART
> >         b.ne    3f
> >         mov     x0, x2
> >         mov     x2, x4
> >         mov     x4, x1
> >         mov     x1, x3
> >         br      x4                              // no return
> >
> > 3:      cmp     x0, #SMC_RESET_VECTORS
> >         beq     9f                              // Nothing to reset!
> >
> >         ldr     x0, =SMC_STUB_ERR
> >         eret
> >
> > 9:      mov     x0, xzr
> >         eret
> > ENDPROC(el1_sync)
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Marc Zyngier]

On Tue, 01 Jun 2021 10:53:49 +0100,
Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> 
> Mark Rutland <mark.rutland@arm.com> 于2021年6月1日周二 下午5:19写道：
> >
> > On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> > > Hi All,
> > >       when Linux kernel boot from EL1,  there is no method to let
> > > kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> > > interface between kernel and EL3 ATF to let kernel can set the
> > > hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> > > shown in [1].
> > > Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> > > when kernel boot from EL1, because the hypervisor vector
> > > table(vbar_el2) is unknown.
> >
> > The kernel already supported being booted at EL2, where it will install
> > itself as the hypervisor (and will drop to EL1 if required). EL2 is the
> > preferred boot mode, as we document in:
> >
> > https://www.kernel.org/doc/html/latest/arm64/booting.html
> >
> > ... where we say:
> >
> > | The CPU must be in either EL2 (RECOMMENDED in order to have access to
> > | the virtualisation extensions) or non-secure EL1.
> >
> > We *strongly* prefer this over adding new ABIs to transition from EL1 to
> > EL2. Please boot the kernel at EL2 if you want to use KVM.
> 
> Thanks for the answer.
> If use KVM, it should boot from EL2.  But if the hypervisor is not
> KVM, such as Jailhouse hypervisor and some Chip manufacturer boot the
> host kernel from EL1(not follow above rule), it seems there is not way
> to enter the Jailhouse hypervisor.

We only deal with two cases:
- either the kernel uses its own, built-in hypervisor: it boots at
  EL2, and installs itself.

- or there is a pre-existing hypervisor, and the kernel boots at EL1.

In the past, Jailhouse used the exact same entry points as KVM. What
has changed?

Finally, if you can change the firmware to install the EL2 vectors,
you can also change it to enter the kernel at EL2. I suggest you do
that instead.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Dongjiu Geng]

Marc Zyngier <maz@kernel.org> 于2021年6月1日周二 下午6:09写道：
>
> On Tue, 01 Jun 2021 10:53:49 +0100,
> Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> >
> > Mark Rutland <mark.rutland@arm.com> 于2021年6月1日周二 下午5:19写道：
> > >
> > > On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> > > > Hi All,
> > > >       when Linux kernel boot from EL1,  there is no method to let
> > > > kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> > > > interface between kernel and EL3 ATF to let kernel can set the
> > > > hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> > > > shown in [1].
> > > > Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> > > > when kernel boot from EL1, because the hypervisor vector
> > > > table(vbar_el2) is unknown.
> > >
> > > The kernel already supported being booted at EL2, where it will install
> > > itself as the hypervisor (and will drop to EL1 if required). EL2 is the
> > > preferred boot mode, as we document in:
> > >
> > > https://www.kernel.org/doc/html/latest/arm64/booting.html
> > >
> > > ... where we say:
> > >
> > > | The CPU must be in either EL2 (RECOMMENDED in order to have access to
> > > | the virtualisation extensions) or non-secure EL1.
> > >
> > > We *strongly* prefer this over adding new ABIs to transition from EL1 to
> > > EL2. Please boot the kernel at EL2 if you want to use KVM.
> >
> > Thanks for the answer.
> > If use KVM, it should boot from EL2.  But if the hypervisor is not
> > KVM, such as Jailhouse hypervisor and some Chip manufacturer boot the
> > host kernel from EL1(not follow above rule), it seems there is not way
> > to enter the Jailhouse hypervisor.
>
> We only deal with two cases:
> - either the kernel uses its own, built-in hypervisor: it boots at
>   EL2, and installs itself.
>
> - or there is a pre-existing hypervisor, and the kernel boots at EL1.
>
> In the past, Jailhouse used the exact same entry points as KVM. What
> has changed?

   Jailhouse use the  __hyp_stub_vectors vector table[1] in linux
kernel arch/arm64/kernel/hyp-stub.S to re-set his own's hypervisor
vector table, but if linux kernel is boot from EL1，it can not use the
entry points(__hyp_stub_vectors). I agree Linux kernel is recommended
boot from EL2, but some custer's boards not follow this rule.

[1]:
ENTRY(__hyp_stub_vectors)
        ventry  el2_sync_invalid                // Synchronous EL2t
        ventry  el2_irq_invalid                 // IRQ EL2t
        ventry  el2_fiq_invalid                 // FIQ EL2t
        ventry  el2_error_invalid               // Error EL2t

        ventry  el2_sync_invalid                // Synchronous EL2h
        ventry  el2_irq_invalid                 // IRQ EL2h
        ventry  el2_fiq_invalid                 // FIQ EL2h
        ventry  el2_error_invalid               // Error EL2h

        ventry  el1_sync                        // Synchronous 64-bit EL1
        ventry  el1_irq_invalid                 // IRQ 64-bit EL1
        ventry  el1_fiq_invalid                 // FIQ 64-bit EL1
        ventry  el1_error_invalid               // Error 64-bit EL1

        ventry  el1_sync_invalid                // Synchronous 32-bit EL1
        ventry  el1_irq_invalid                 // IRQ 32-bit EL1
        ventry  el1_fiq_invalid                 // FIQ 32-bit EL1
        ventry  el1_error_invalid               // Error 32-bit EL1
ENDPROC(__hyp_stub_vectors)

>
> Finally, if you can change the firmware to install the EL2 vectors,
> you can also change it to enter the kernel at EL2. I suggest you do
> that instead.

I agree with you,  but needs to change customer‘s board, I will try to
discuess with customer.


>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Marc Zyngier]

On Tue, 01 Jun 2021 14:34:05 +0100,
Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> 
> Marc Zyngier <maz@kernel.org> 于2021年6月1日周二 下午6:09写道：
> >
> > On Tue, 01 Jun 2021 10:53:49 +0100,
> > Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> > >
> > > Mark Rutland <mark.rutland@arm.com> 于2021年6月1日周二 下午5:19写道：
> > > >
> > > > On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> > > > > Hi All,
> > > > >       when Linux kernel boot from EL1,  there is no method to let
> > > > > kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> > > > > interface between kernel and EL3 ATF to let kernel can set the
> > > > > hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> > > > > shown in [1].
> > > > > Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> > > > > when kernel boot from EL1, because the hypervisor vector
> > > > > table(vbar_el2) is unknown.
> > > >
> > > > The kernel already supported being booted at EL2, where it will install
> > > > itself as the hypervisor (and will drop to EL1 if required). EL2 is the
> > > > preferred boot mode, as we document in:
> > > >
> > > > https://www.kernel.org/doc/html/latest/arm64/booting.html
> > > >
> > > > ... where we say:
> > > >
> > > > | The CPU must be in either EL2 (RECOMMENDED in order to have access to
> > > > | the virtualisation extensions) or non-secure EL1.
> > > >
> > > > We *strongly* prefer this over adding new ABIs to transition from EL1 to
> > > > EL2. Please boot the kernel at EL2 if you want to use KVM.
> > >
> > > Thanks for the answer.
> > > If use KVM, it should boot from EL2.  But if the hypervisor is not
> > > KVM, such as Jailhouse hypervisor and some Chip manufacturer boot the
> > > host kernel from EL1(not follow above rule), it seems there is not way
> > > to enter the Jailhouse hypervisor.
> >
> > We only deal with two cases:
> > - either the kernel uses its own, built-in hypervisor: it boots at
> >   EL2, and installs itself.
> >
> > - or there is a pre-existing hypervisor, and the kernel boots at EL1.
> >
> > In the past, Jailhouse used the exact same entry points as KVM. What
> > has changed?
> 
>    Jailhouse use the  __hyp_stub_vectors vector table[1] in linux
> kernel arch/arm64/kernel/hyp-stub.S to re-set his own's hypervisor
> vector table, but if linux kernel is boot from EL1，it can not use the
> entry points(__hyp_stub_vectors). I agree Linux kernel is recommended
> boot from EL2, but some custer's boards not follow this rule.

So Jailhouse doesn't have this problem when used as intended.

> 
> [1]:
> ENTRY(__hyp_stub_vectors)
>         ventry  el2_sync_invalid                // Synchronous EL2t
>         ventry  el2_irq_invalid                 // IRQ EL2t
>         ventry  el2_fiq_invalid                 // FIQ EL2t
>         ventry  el2_error_invalid               // Error EL2t
> 
>         ventry  el2_sync_invalid                // Synchronous EL2h
>         ventry  el2_irq_invalid                 // IRQ EL2h
>         ventry  el2_fiq_invalid                 // FIQ EL2h
>         ventry  el2_error_invalid               // Error EL2h
> 
>         ventry  el1_sync                        // Synchronous 64-bit EL1
>         ventry  el1_irq_invalid                 // IRQ 64-bit EL1
>         ventry  el1_fiq_invalid                 // FIQ 64-bit EL1
>         ventry  el1_error_invalid               // Error 64-bit EL1
> 
>         ventry  el1_sync_invalid                // Synchronous 32-bit EL1
>         ventry  el1_irq_invalid                 // IRQ 32-bit EL1
>         ventry  el1_fiq_invalid                 // FIQ 32-bit EL1
>         ventry  el1_error_invalid               // Error 32-bit EL1
> ENDPROC(__hyp_stub_vectors)
> 
> >
> > Finally, if you can change the firmware to install the EL2 vectors,
> > you can also change it to enter the kernel at EL2. I suggest you do
> > that instead.
> 
> I agree with you,  but needs to change customer‘s board, I will try to
> discuess with customer.

In both cases, you'll need to change your customer's firmware.

It seems to me that there is no reason for the arm64 boot protocol to
change and adopt weird, wonderful and proprietary privilege escalation
methods.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Linux kernel set the hypervisor vector table through ATF

[Dongjiu Geng]

Marc Zyngier <maz@kernel.org> 于2021年6月1日周二 下午9:43写道：
>
> On Tue, 01 Jun 2021 14:34:05 +0100,
> Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> >
> > Marc Zyngier <maz@kernel.org> 于2021年6月1日周二 下午6:09写道：
> > >
> > > On Tue, 01 Jun 2021 10:53:49 +0100,
> > > Dongjiu Geng <gengdongjiu1@gmail.com> wrote:
> > > >
> > > > Mark Rutland <mark.rutland@arm.com> 于2021年6月1日周二 下午5:19写道：
> > > > >
> > > > > On Fri, May 28, 2021 at 05:26:51PM +0800, Dongjiu Geng wrote:
> > > > > > Hi All,
> > > > > >       when Linux kernel boot from EL1,  there is no method to let
> > > > > > kernel to enter EL2 to enable hypervisor.  so I want to add an SMC
> > > > > > interface between kernel and EL3 ATF to let kernel can set the
> > > > > > hypervisor vector table,  then can enter EL2 to enable hypervisor, as
> > > > > > shown in [1].
> > > > > > Do you agree?  Otherwise there is no method to enter EL2 hypervisor
> > > > > > when kernel boot from EL1, because the hypervisor vector
> > > > > > table(vbar_el2) is unknown.
> > > > >
> > > > > The kernel already supported being booted at EL2, where it will install
> > > > > itself as the hypervisor (and will drop to EL1 if required). EL2 is the
> > > > > preferred boot mode, as we document in:
> > > > >
> > > > > https://www.kernel.org/doc/html/latest/arm64/booting.html
> > > > >
> > > > > ... where we say:
> > > > >
> > > > > | The CPU must be in either EL2 (RECOMMENDED in order to have access to
> > > > > | the virtualisation extensions) or non-secure EL1.
> > > > >
> > > > > We *strongly* prefer this over adding new ABIs to transition from EL1 to
> > > > > EL2. Please boot the kernel at EL2 if you want to use KVM.
> > > >
> > > > Thanks for the answer.
> > > > If use KVM, it should boot from EL2.  But if the hypervisor is not
> > > > KVM, such as Jailhouse hypervisor and some Chip manufacturer boot the
> > > > host kernel from EL1(not follow above rule), it seems there is not way
> > > > to enter the Jailhouse hypervisor.
> > >
> > > We only deal with two cases:
> > > - either the kernel uses its own, built-in hypervisor: it boots at
> > >   EL2, and installs itself.
> > >
> > > - or there is a pre-existing hypervisor, and the kernel boots at EL1.
> > >
> > > In the past, Jailhouse used the exact same entry points as KVM. What
> > > has changed?
> >
> >    Jailhouse use the  __hyp_stub_vectors vector table[1] in linux
> > kernel arch/arm64/kernel/hyp-stub.S to re-set his own's hypervisor
> > vector table, but if linux kernel is boot from EL1，it can not use the
> > entry points(__hyp_stub_vectors). I agree Linux kernel is recommended
> > boot from EL2, but some custer's boards not follow this rule.
>
> So Jailhouse doesn't have this problem when used as intended.
>
> >
> > [1]:
> > ENTRY(__hyp_stub_vectors)
> >         ventry  el2_sync_invalid                // Synchronous EL2t
> >         ventry  el2_irq_invalid                 // IRQ EL2t
> >         ventry  el2_fiq_invalid                 // FIQ EL2t
> >         ventry  el2_error_invalid               // Error EL2t
> >
> >         ventry  el2_sync_invalid                // Synchronous EL2h
> >         ventry  el2_irq_invalid                 // IRQ EL2h
> >         ventry  el2_fiq_invalid                 // FIQ EL2h
> >         ventry  el2_error_invalid               // Error EL2h
> >
> >         ventry  el1_sync                        // Synchronous 64-bit EL1
> >         ventry  el1_irq_invalid                 // IRQ 64-bit EL1
> >         ventry  el1_fiq_invalid                 // FIQ 64-bit EL1
> >         ventry  el1_error_invalid               // Error 64-bit EL1
> >
> >         ventry  el1_sync_invalid                // Synchronous 32-bit EL1
> >         ventry  el1_irq_invalid                 // IRQ 32-bit EL1
> >         ventry  el1_fiq_invalid                 // FIQ 32-bit EL1
> >         ventry  el1_error_invalid               // Error 32-bit EL1
> > ENDPROC(__hyp_stub_vectors)
> >
> > >
> > > Finally, if you can change the firmware to install the EL2 vectors,
> > > you can also change it to enter the kernel at EL2. I suggest you do
> > > that instead.
> >
> > I agree with you,  but needs to change customer‘s board, I will try to
> > discuess with customer.
>
> In both cases, you'll need to change your customer's firmware.
>
> It seems to me that there is no reason for the arm64 boot protocol to
> change and adopt weird, wonderful and proprietary privilege escalation
> methods.

Got it, thanks for the answer.

>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel