Xen ARM - Exposing a PL011 to the guest

Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi all,

Few months ago, Linaro has published the version 2 of the VM 
specification [1].

For those who don't know, the specification provides guidelines to 
guarantee a compliant OS images could run on various hypervisor (e.g 
Xen, KVM).

Looking at the specification, it will require Xen to expose new devices 
to the guest: pl011, rtc, persistent flash (for UEFI variables).

The RTC and persistent will only be used by the UEFI firwmare. The 
firwmare is custom made for Xen guest and be loaded by the toolstack, so 
we could theoretically provide PV drivers for those.

This is not the case for the PL011. The guest will be shipped with a 
PL011/SBSA UART driver,.This means it will expect to access it through MMIO.

So we have to emulate a PL011. The question is where? Before suggesting 
some ideas, the guest/user will expect to be able to interact with the 
console through the UART. This means that the UART and xenconsoled needs 
to communicate together.

I think we can distinct two places where the PL011 could be emulated:
in the hypervisor, or outside the hypervisor.

Emulating the UART in the hypervisor means that we take the risk to 
increase to the attack surface of Xen if there is a bug in the emulation 
code. The attack surface could be reduced by emulating the UART in 
another exception level (e.g EL1, EL0) but still under the control of 
the hypervisor. Usually the guest is communicating between with 
xenconsoled using a ring. For the first console this could be discovered 
using hypercall HVMOP_get_param. For the second and onwards, it 
described in xenstore. I would not worry too much about emulating 
multiple PL011s, so we could implement the PV frontend in Xen.

Emulating the UART outside the hypervisor (e.g in DOM0 or special 
domain) would require to bring the concept of ioreq server on ARM. Which 
left the question where do we emulate the PL011? The best place would be 
xenconsoled. But I am not sure how would be the security impact here. 
Does all guest consoles are emulated within the same daemon?

I would lean towards the first solution if we implement all the security 
safety I mentioned. Although, the second solution would be a good move 
if we decide to implement more devices (e.g RTC, pflash) in the future.

Do you have any opinions?

Cheers,

[1] 
http://people.linaro.org/~christoffer.dall/VMSystemSpecificationForARM-v2.0-rc1.pdf

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Wed, Nov 30, 2016 at 03:29:32PM +0000, Julien Grall wrote:
> Hi all,
> 
> Few months ago, Linaro has published the version 2 of the VM
> specification [1].
> 
> For those who don't know, the specification provides guidelines to
> guarantee a compliant OS images could run on various hypervisor (e.g
> Xen, KVM).
> 
> Looking at the specification, it will require Xen to expose new
> devices to the guest: pl011, rtc, persistent flash (for UEFI
> variables).
> 
> The RTC and persistent will only be used by the UEFI firwmare. 

Why would a guest booting without UEFI not want to use the RTC directly?

Linux does this on KVM today...

-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi Christoffer,

On 30/11/16 16:24, Christoffer Dall wrote:
> On Wed, Nov 30, 2016 at 03:29:32PM +0000, Julien Grall wrote:
>> Hi all,
>>
>> Few months ago, Linaro has published the version 2 of the VM
>> specification [1].
>>
>> For those who don't know, the specification provides guidelines to
>> guarantee a compliant OS images could run on various hypervisor (e.g
>> Xen, KVM).
>>
>> Looking at the specification, it will require Xen to expose new
>> devices to the guest: pl011, rtc, persistent flash (for UEFI
>> variables).
>>
>> The RTC and persistent will only be used by the UEFI firwmare.
>
> Why would a guest booting without UEFI not want to use the RTC directly?

Sorry, I had only the VM spec in mind. Of course, an RTC could be used 
for guest booting without UEFI.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Volodymyr Babchuk]

Hello Julien,



On 30 November 2016 at 17:29, Julien Grall <julien.grall@arm.com> wrote:
[...]
> I think we can distinct two places where the PL011 could be emulated:
> in the hypervisor, or outside the hypervisor.
>
> Emulating the UART in the hypervisor means that we take the risk to increase
> to the attack surface of Xen if there is a bug in the emulation code. The
> attack surface could be reduced by emulating the UART in another exception
> level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
> the guest is communicating between with xenconsoled using a ring. For the
> first console this could be discovered using hypercall HVMOP_get_param. For
> the second and onwards, it described in xenstore. I would not worry too much
> about emulating multiple PL011s, so we could implement the PV frontend in
> Xen.
>
[...]

> I would lean towards the first solution if we implement all the security
> safety I mentioned. Although, the second solution would be a good move if we
> decide to implement more devices (e.g RTC, pflash) in the future.
>
> Do you have any opinions?
Looks like this topic have some in common with OP-TEE thread. I like
first solution, because if there will be easy and reliable way to run
code in XEN's EL1/EL0, then this will be ideal solution for TEE
emulation/mediation layer.
So, if you'll choose this way, please bear in mind other uses, like
TEE emulation.

-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Wed, 30 Nov 2016, Julien Grall wrote:
> Hi all,
> 
> Few months ago, Linaro has published the version 2 of the VM specification
> [1].
> 
> For those who don't know, the specification provides guidelines to guarantee a
> compliant OS images could run on various hypervisor (e.g Xen, KVM).
> 
> Looking at the specification, it will require Xen to expose new devices to the
> guest: pl011, rtc, persistent flash (for UEFI variables).
> 
> The RTC and persistent will only be used by the UEFI firwmare. The firwmare is
> custom made for Xen guest and be loaded by the toolstack, so we could
> theoretically provide PV drivers for those.
> 
> This is not the case for the PL011. The guest will be shipped with a
> PL011/SBSA UART driver,.This means it will expect to access it through MMIO.
> 
> So we have to emulate a PL011. The question is where? Before suggesting some
> ideas, the guest/user will expect to be able to interact with the console
> through the UART. This means that the UART and xenconsoled needs to
> communicate together.
> 
> I think we can distinct two places where the PL011 could be emulated:
> in the hypervisor, or outside the hypervisor.
> 
> Emulating the UART in the hypervisor means that we take the risk to increase
> to the attack surface of Xen if there is a bug in the emulation code. The
> attack surface could be reduced by emulating the UART in another exception
> level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
> the guest is communicating between with xenconsoled using a ring. For the
> first console this could be discovered using hypercall HVMOP_get_param. For
> the second and onwards, it described in xenstore. I would not worry too much
> about emulating multiple PL011s, so we could implement the PV frontend in Xen.
> 
> Emulating the UART outside the hypervisor (e.g in DOM0 or special domain)
> would require to bring the concept of ioreq server on ARM. Which left the
> question where do we emulate the PL011? The best place would be xenconsoled.
> But I am not sure how would be the security impact here. Does all guest
> consoles are emulated within the same daemon?

One xenconsoled instance handles all PV console frontends. However QEMU,
not xenconsoled, handles secondary PV consoles and emulated serials. One
QEMU instance per VM. The PV console protocol is pretty trivial and not
easy to exploit.

Instead of emulating PL011 in xenconsoled we could do it in QEMU (or
something like it). But I think we should do something else instead, see
below.


> I would lean towards the first solution if we implement all the security
> safety I mentioned. Although, the second solution would be a good move if we
> decide to implement more devices (e.g RTC, pflash) in the future.
> 
> Do you have any opinions?

As I have just written in this other email:

http://marc.info/?l=xen-devel&m=148054285829397

I don't think we should introduce userspace emulators in Dom0 on ARM.
The reason is that they are bad both for security and performance. In
general I think that the best solution is to introduce emulators in EL1
in Xen.

That said, PL011 is pretty small and the PL011 emulator needs to feed
data back to Dom0 for user interaction, so, if we need to, we could make
an exception for it and run it in userspace Dom0.

But do we really need to make an exception? I don't think so. The PL011
emulator could run in EL1 in Xen and connect to xenconsoled using the
regular PV console frontend/backend protocol, as you suggested. One
caveat: I would avoid introducing xenstore support in Xen, but
fortunately we don't need xenstore support for the first PV console
connection, so we should be OK. In any case if the xenconsole protocol
or xenconsoled are the problem, we could easily rewrite them or add
specific support for this new use case.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote:
> On Wed, 30 Nov 2016, Julien Grall wrote:
> > Hi all,
> > 
> > Few months ago, Linaro has published the version 2 of the VM specification
> > [1].
> > 
> > For those who don't know, the specification provides guidelines to guarantee a
> > compliant OS images could run on various hypervisor (e.g Xen, KVM).
> > 
> > Looking at the specification, it will require Xen to expose new devices to the
> > guest: pl011, rtc, persistent flash (for UEFI variables).
> > 
> > The RTC and persistent will only be used by the UEFI firwmare. The firwmare is
> > custom made for Xen guest and be loaded by the toolstack, so we could
> > theoretically provide PV drivers for those.
> > 
> > This is not the case for the PL011. The guest will be shipped with a
> > PL011/SBSA UART driver,.This means it will expect to access it through MMIO.
> > 
> > So we have to emulate a PL011. The question is where? Before suggesting some
> > ideas, the guest/user will expect to be able to interact with the console
> > through the UART. This means that the UART and xenconsoled needs to
> > communicate together.
> > 
> > I think we can distinct two places where the PL011 could be emulated:
> > in the hypervisor, or outside the hypervisor.
> > 
> > Emulating the UART in the hypervisor means that we take the risk to increase
> > to the attack surface of Xen if there is a bug in the emulation code. The
> > attack surface could be reduced by emulating the UART in another exception
> > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
> > the guest is communicating between with xenconsoled using a ring. For the
> > first console this could be discovered using hypercall HVMOP_get_param. For
> > the second and onwards, it described in xenstore. I would not worry too much
> > about emulating multiple PL011s, so we could implement the PV frontend in Xen.
> > 
> > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain)
> > would require to bring the concept of ioreq server on ARM. Which left the
> > question where do we emulate the PL011? The best place would be xenconsoled.
> > But I am not sure how would be the security impact here. Does all guest
> > consoles are emulated within the same daemon?
> 
> One xenconsoled instance handles all PV console frontends. However QEMU,
> not xenconsoled, handles secondary PV consoles and emulated serials. One
> QEMU instance per VM. The PV console protocol is pretty trivial and not
> easy to exploit.
> 
> Instead of emulating PL011 in xenconsoled we could do it in QEMU (or
> something like it). But I think we should do something else instead, see
> below.
> 
> 
> > I would lean towards the first solution if we implement all the security
> > safety I mentioned. Although, the second solution would be a good move if we
> > decide to implement more devices (e.g RTC, pflash) in the future.
> > 
> > Do you have any opinions?
> 
> As I have just written in this other email:
> 
> http://marc.info/?l=xen-devel&m=148054285829397
> 
> I don't think we should introduce userspace emulators in Dom0 on ARM.
> The reason is that they are bad both for security and performance. In
> general I think that the best solution is to introduce emulators in EL1
> in Xen.

Slightly hijacking the discussion:

Why would you run stuff in EL1 as opposed to EL0?

My understanding of the HCR_EL2.TGE bit is exactly designed to run
bare-metal applications on top of a hypervisor running in EL2 and has
the added benefit that I think you can switch to your bare-metal
application without having to context-switch any EL1 tate, because TGE
just disabled the whole stage 1 MMU.  Obviously this doesn't give you
virtual memory, unless you map that using stage 2 tables, but that might
actually be preferred, and since the whole point is to separate your
non-privileged hypervisor applications from the hypervisor's memory
anyhow, it may be fine.

I don't know enough about Xen's impementation to say whether this is
doable or not, but I think there are potential advantages in terms of
performance and using the architecture as designed.

Thinking about this, if you run something in EL1 on top of Xen without
setting the TGE bit, what happens if your application misbehaves, for
example causes a page fault?  That would be taken at EL1, right?  So
will you install an EL1 handler for exceptions that somehow forwards the
trap into EL2?  Sounds cleaner to me to catch anything weird the
application does to EL2.

-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Thu, 1 Dec 2016, Christoffer Dall wrote:
> On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote:
> > On Wed, 30 Nov 2016, Julien Grall wrote:
> > > Hi all,
> > > 
> > > Few months ago, Linaro has published the version 2 of the VM specification
> > > [1].
> > > 
> > > For those who don't know, the specification provides guidelines to guarantee a
> > > compliant OS images could run on various hypervisor (e.g Xen, KVM).
> > > 
> > > Looking at the specification, it will require Xen to expose new devices to the
> > > guest: pl011, rtc, persistent flash (for UEFI variables).
> > > 
> > > The RTC and persistent will only be used by the UEFI firwmare. The firwmare is
> > > custom made for Xen guest and be loaded by the toolstack, so we could
> > > theoretically provide PV drivers for those.
> > > 
> > > This is not the case for the PL011. The guest will be shipped with a
> > > PL011/SBSA UART driver,.This means it will expect to access it through MMIO.
> > > 
> > > So we have to emulate a PL011. The question is where? Before suggesting some
> > > ideas, the guest/user will expect to be able to interact with the console
> > > through the UART. This means that the UART and xenconsoled needs to
> > > communicate together.
> > > 
> > > I think we can distinct two places where the PL011 could be emulated:
> > > in the hypervisor, or outside the hypervisor.
> > > 
> > > Emulating the UART in the hypervisor means that we take the risk to increase
> > > to the attack surface of Xen if there is a bug in the emulation code. The
> > > attack surface could be reduced by emulating the UART in another exception
> > > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
> > > the guest is communicating between with xenconsoled using a ring. For the
> > > first console this could be discovered using hypercall HVMOP_get_param. For
> > > the second and onwards, it described in xenstore. I would not worry too much
> > > about emulating multiple PL011s, so we could implement the PV frontend in Xen.
> > > 
> > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain)
> > > would require to bring the concept of ioreq server on ARM. Which left the
> > > question where do we emulate the PL011? The best place would be xenconsoled.
> > > But I am not sure how would be the security impact here. Does all guest
> > > consoles are emulated within the same daemon?
> > 
> > One xenconsoled instance handles all PV console frontends. However QEMU,
> > not xenconsoled, handles secondary PV consoles and emulated serials. One
> > QEMU instance per VM. The PV console protocol is pretty trivial and not
> > easy to exploit.
> > 
> > Instead of emulating PL011 in xenconsoled we could do it in QEMU (or
> > something like it). But I think we should do something else instead, see
> > below.
> > 
> > 
> > > I would lean towards the first solution if we implement all the security
> > > safety I mentioned. Although, the second solution would be a good move if we
> > > decide to implement more devices (e.g RTC, pflash) in the future.
> > > 
> > > Do you have any opinions?
> > 
> > As I have just written in this other email:
> > 
> > http://marc.info/?l=xen-devel&m=148054285829397
> > 
> > I don't think we should introduce userspace emulators in Dom0 on ARM.
> > The reason is that they are bad both for security and performance. In
> > general I think that the best solution is to introduce emulators in EL1
> > in Xen.
> 
> Slightly hijacking the discussion:
> 
> Why would you run stuff in EL1 as opposed to EL0?
> 
> My understanding of the HCR_EL2.TGE bit is exactly designed to run
> bare-metal applications on top of a hypervisor running in EL2 and has
> the added benefit that I think you can switch to your bare-metal
> application without having to context-switch any EL1 tate, because TGE
> just disabled the whole stage 1 MMU.  Obviously this doesn't give you
> virtual memory, unless you map that using stage 2 tables, but that might
> actually be preferred, and since the whole point is to separate your
> non-privileged hypervisor applications from the hypervisor's memory
> anyhow, it may be fine.
> 
> I don't know enough about Xen's impementation to say whether this is
> doable or not, but I think there are potential advantages in terms of
> performance and using the architecture as designed.

You might be right and it is definitely worth investigating.


> Thinking about this, if you run something in EL1 on top of Xen without
> setting the TGE bit, what happens if your application misbehaves, for
> example causes a page fault?  That would be taken at EL1, right?  So
> will you install an EL1 handler for exceptions that somehow forwards the
> trap into EL2?  Sounds cleaner to me to catch anything weird the
> application does to EL2.

Yes, I think it would be better to handle that kind of faults at EL2.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,

The idea is for Xen to act as an intermediary as shown below:

               ring buffers
                           rx/tx fifo
dom0 <-------------------> Xen HYP (running pl011 emulation)
<-------------------> domU
                    event
                           interrupts

Xen will directly manage the in/out console ring buffers (allocated by
dom0 for dom0-domU console communication) for reading/writing console
data from/to dom0. On the other side, Xen HYP will emulate pl011 to
read/write data from/to domU and pass it on to/from dom0 over the
in/out console ring buffers. There should be no change in dom0 as it
will still use the same ring buffers. Similarly there should be no
change in domU which would be running a standard pll011 driver.

Currently, I am working on the interface between dom0 and Xen HYP. I
want to intercept the console events in Xen HYP which pass between
dom0 and domU. For now, I just want to capture console data coming
from dom0 at Xen HYP and loop it back to dom0, to confirm that this
interface is working.

Since each guest domain will have a unique event channel assigned for
console communication, Xen HYP can find out the event channel for a
given domU from the start_info page of that domU, which should have
been allocated by dom0. Whenever, an event is to be dispatched via
evtchn_send() API in Xen, it can check if the event channel is the
console event channel for a given domU. If yes and its source domain
is dom0 and destination domain is domU then it will write the data
back to the console out ring buffer of the domU and raise a console
event to dom0.

Once this interface is working, Xen HYP can check the source and
destination dom ids and decide which way the event came from and
accordingly process the console data. To allow a mix of PV console
guests and pl011 guests, Xen might have to maintain a flag per domain,
which tells whether Xen HYP should intercept and process the data (for
pl011 UART case) or let it go transparently (for PV conosle case).

Is this the correct way to find out the console event channel for a
given domU or there is some other way? I could not locate in the Xen
source code, where exactly the start_info page is created for a new
domain, which would contain the console event channel and and machine
page where the ring buffers are mapped.

Regards,
Bhupinder

On 2 December 2016 at 03:25, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Thu, 1 Dec 2016, Christoffer Dall wrote:
>> On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote:
>> > On Wed, 30 Nov 2016, Julien Grall wrote:
>> > > Hi all,
>> > >
>> > > Few months ago, Linaro has published the version 2 of the VM specification
>> > > [1].
>> > >
>> > > For those who don't know, the specification provides guidelines to guarantee a
>> > > compliant OS images could run on various hypervisor (e.g Xen, KVM).
>> > >
>> > > Looking at the specification, it will require Xen to expose new devices to the
>> > > guest: pl011, rtc, persistent flash (for UEFI variables).
>> > >
>> > > The RTC and persistent will only be used by the UEFI firwmare. The firwmare is
>> > > custom made for Xen guest and be loaded by the toolstack, so we could
>> > > theoretically provide PV drivers for those.
>> > >
>> > > This is not the case for the PL011. The guest will be shipped with a
>> > > PL011/SBSA UART driver,.This means it will expect to access it through MMIO.
>> > >
>> > > So we have to emulate a PL011. The question is where? Before suggesting some
>> > > ideas, the guest/user will expect to be able to interact with the console
>> > > through the UART. This means that the UART and xenconsoled needs to
>> > > communicate together.
>> > >
>> > > I think we can distinct two places where the PL011 could be emulated:
>> > > in the hypervisor, or outside the hypervisor.
>> > >
>> > > Emulating the UART in the hypervisor means that we take the risk to increase
>> > > to the attack surface of Xen if there is a bug in the emulation code. The
>> > > attack surface could be reduced by emulating the UART in another exception
>> > > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
>> > > the guest is communicating between with xenconsoled using a ring. For the
>> > > first console this could be discovered using hypercall HVMOP_get_param. For
>> > > the second and onwards, it described in xenstore. I would not worry too much
>> > > about emulating multiple PL011s, so we could implement the PV frontend in Xen.
>> > >
>> > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain)
>> > > would require to bring the concept of ioreq server on ARM. Which left the
>> > > question where do we emulate the PL011? The best place would be xenconsoled.
>> > > But I am not sure how would be the security impact here. Does all guest
>> > > consoles are emulated within the same daemon?
>> >
>> > One xenconsoled instance handles all PV console frontends. However QEMU,
>> > not xenconsoled, handles secondary PV consoles and emulated serials. One
>> > QEMU instance per VM. The PV console protocol is pretty trivial and not
>> > easy to exploit.
>> >
>> > Instead of emulating PL011 in xenconsoled we could do it in QEMU (or
>> > something like it). But I think we should do something else instead, see
>> > below.
>> >
>> >
>> > > I would lean towards the first solution if we implement all the security
>> > > safety I mentioned. Although, the second solution would be a good move if we
>> > > decide to implement more devices (e.g RTC, pflash) in the future.
>> > >
>> > > Do you have any opinions?
>> >
>> > As I have just written in this other email:
>> >
>> > http://marc.info/?l=xen-devel&m=148054285829397
>> >
>> > I don't think we should introduce userspace emulators in Dom0 on ARM.
>> > The reason is that they are bad both for security and performance. In
>> > general I think that the best solution is to introduce emulators in EL1
>> > in Xen.
>>
>> Slightly hijacking the discussion:
>>
>> Why would you run stuff in EL1 as opposed to EL0?
>>
>> My understanding of the HCR_EL2.TGE bit is exactly designed to run
>> bare-metal applications on top of a hypervisor running in EL2 and has
>> the added benefit that I think you can switch to your bare-metal
>> application without having to context-switch any EL1 tate, because TGE
>> just disabled the whole stage 1 MMU.  Obviously this doesn't give you
>> virtual memory, unless you map that using stage 2 tables, but that might
>> actually be preferred, and since the whole point is to separate your
>> non-privileged hypervisor applications from the hypervisor's memory
>> anyhow, it may be fine.
>>
>> I don't know enough about Xen's impementation to say whether this is
>> doable or not, but I think there are potential advantages in terms of
>> performance and using the architecture as designed.
>
> You might be right and it is definitely worth investigating.
>
>
>> Thinking about this, if you run something in EL1 on top of Xen without
>> setting the TGE bit, what happens if your application misbehaves, for
>> example causes a page fault?  That would be taken at EL1, right?  So
>> will you install an EL1 handler for exceptions that somehow forwards the
>> trap into EL2?  Sounds cleaner to me to catch anything weird the
>> application does to EL2.
>
> Yes, I think it would be better to handle that kind of faults at EL2.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

(CC rest maintainers for event channel questions)

On 16/12/16 10:06, Bhupinder Thakur wrote:
> Hi,

Hi Bhupinder,

> The idea is for Xen to act as an intermediary as shown below:
>
>                ring buffers
>                            rx/tx fifo
> dom0 <-------------------> Xen HYP (running pl011 emulation)
> <-------------------> domU
>                     event
>                            interrupts
>
> Xen will directly manage the in/out console ring buffers (allocated by
> dom0 for dom0-domU console communication) for reading/writing console
> data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> read/write data from/to domU and pass it on to/from dom0 over the
> in/out console ring buffers. There should be no change in dom0 as it
> will still use the same ring buffers. Similarly there should be no
> change in domU which would be running a standard pll011 driver.
>
> Currently, I am working on the interface between dom0 and Xen HYP. I
> want to intercept the console events in Xen HYP which pass between
> dom0 and domU. For now, I just want to capture console data coming
> from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> interface is working.
>
> Since each guest domain will have a unique event channel assigned for
> console communication, Xen HYP can find out the event channel for a
> given domU from the start_info page of that domU, which should have

The start_info page is x86 specific. If you want to get the console 
event channel for ARM, you would have to use 
d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].

This parameter will be setup by the toolstack (see alloc_magic_pages in 
libxc/xc_dom_arm.c).

> been allocated by dom0. Whenever, an event is to be dispatched via
> evtchn_send() API in Xen, it can check if the event channel is the
> console event channel for a given domU. If yes and its source domain
> is dom0 and destination domain is domU then it will write the data
> back to the console out ring buffer of the domU and raise a console
> event to dom0.
>
> Once this interface is working, Xen HYP can check the source and
> destination dom ids and decide which way the event came from and
> accordingly process the console data. To allow a mix of PV console
> guests and pl011 guests, Xen might have to maintain a flag per domain,
> which tells whether Xen HYP should intercept and process the data (for
> pl011 UART case) or let it go transparently (for PV conosle case).

I am not very familiar with the event channel code. I will let the 
others comment on this bit.

Regardless that, how would you decide whether the hypervisor should 
intercept the notification?

I can see 2 different cases:
	1) The guest is starting to use the pl011 then move to the HVC console 
(or HVC then pl011)
	2) The guest is using both the PL011 and the HVC console

Should we consider the second case valid? I would say yes, because a 
user could specify both on the command line. If we use the same ring, 
the output would be a total garbage.

So maybe we need to allocate two distinct rings and event channel?

For your testing, you could consider that the ring is only used by the 
pl011. I would recommend you to disable the HVC console driver in the 
guest (see CONFIG_HVC_XEN).

I looked at the event channel code and I can only find a function for 
xen to create an event channel (see alloc_unbound_xen_event_channel). 
But in your case, you want a function to bind an event channel.

>
> Is this the correct way to find out the console event channel for a
> given domU or there is some other way? I could not locate in the Xen
> source code, where exactly the start_info page is created for a new
> domain, which would contain the console event channel and and machine
> page where the ring buffers are mapped.

You can give a look to HVM_PARAM_CONSOLE_EVTCHN and 
HVM_PARAM_CONSOLE_PFN (see more details above).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> (CC rest maintainers for event channel questions)
> 
> On 16/12/16 10:06, Bhupinder Thakur wrote:
> >Hi,
> 
> Hi Bhupinder,
> 
> >The idea is for Xen to act as an intermediary as shown below:
> >
> >               ring buffers
> >                           rx/tx fifo
> >dom0 <-------------------> Xen HYP (running pl011 emulation)
> ><-------------------> domU
> >                    event
> >                           interrupts
> >
> >Xen will directly manage the in/out console ring buffers (allocated by
> >dom0 for dom0-domU console communication) for reading/writing console
> >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> >read/write data from/to domU and pass it on to/from dom0 over the
> >in/out console ring buffers. There should be no change in dom0 as it
> >will still use the same ring buffers. Similarly there should be no
> >change in domU which would be running a standard pll011 driver.
> >
> >Currently, I am working on the interface between dom0 and Xen HYP. I
> >want to intercept the console events in Xen HYP which pass between
> >dom0 and domU. For now, I just want to capture console data coming
> >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> >interface is working.
> >
> >Since each guest domain will have a unique event channel assigned for
> >console communication, Xen HYP can find out the event channel for a
> >given domU from the start_info page of that domU, which should have
> 
> The start_info page is x86 specific. If you want to get the console
> event channel for ARM, you would have to use
> d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> 
> This parameter will be setup by the toolstack (see alloc_magic_pages
> in libxc/xc_dom_arm.c).
> 
> >been allocated by dom0. Whenever, an event is to be dispatched via
> >evtchn_send() API in Xen, it can check if the event channel is the
> >console event channel for a given domU. If yes and its source domain
> >is dom0 and destination domain is domU then it will write the data
> >back to the console out ring buffer of the domU and raise a console
> >event to dom0.
> >
> >Once this interface is working, Xen HYP can check the source and
> >destination dom ids and decide which way the event came from and
> >accordingly process the console data. To allow a mix of PV console
> >guests and pl011 guests, Xen might have to maintain a flag per domain,
> >which tells whether Xen HYP should intercept and process the data (for
> >pl011 UART case) or let it go transparently (for PV conosle case).
> 
> I am not very familiar with the event channel code. I will let the
> others comment on this bit.
> 
> Regardless that, how would you decide whether the hypervisor should
> intercept the notification?
> 
> I can see 2 different cases:
> 	1) The guest is starting to use the pl011 then move to the HVC
> console (or HVC then pl011)
> 	2) The guest is using both the PL011 and the HVC console
> 
> Should we consider the second case valid? I would say yes, because a
> user could specify both on the command line. If we use the same
> ring, the output would be a total garbage.
> 
> So maybe we need to allocate two distinct rings and event channel?

This sounds like the only sensible thing to me.  I think this is really
about adding a new device to the Xen virtual platform, and providing the
user the option to choose which one he wants the tool in Dom0 to be
presented using stdin/out.  Presumably the other console/serial can be
redirected to a file or socket or something?


Thanks,
-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Mon, 19 Dec 2016, Christoffer Dall wrote:
> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > (CC rest maintainers for event channel questions)
> > 
> > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > >Hi,
> > 
> > Hi Bhupinder,
> > 
> > >The idea is for Xen to act as an intermediary as shown below:
> > >
> > >               ring buffers
> > >                           rx/tx fifo
> > >dom0 <-------------------> Xen HYP (running pl011 emulation)
> > ><-------------------> domU
> > >                    event
> > >                           interrupts
> > >
> > >Xen will directly manage the in/out console ring buffers (allocated by
> > >dom0 for dom0-domU console communication) for reading/writing console
> > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > >read/write data from/to domU and pass it on to/from dom0 over the
> > >in/out console ring buffers. There should be no change in dom0 as it
> > >will still use the same ring buffers. Similarly there should be no
> > >change in domU which would be running a standard pll011 driver.
> > >
> > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > >want to intercept the console events in Xen HYP which pass between
> > >dom0 and domU. For now, I just want to capture console data coming
> > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > >interface is working.
> > >
> > >Since each guest domain will have a unique event channel assigned for
> > >console communication, Xen HYP can find out the event channel for a
> > >given domU from the start_info page of that domU, which should have
> > 
> > The start_info page is x86 specific. If you want to get the console
> > event channel for ARM, you would have to use
> > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > 
> > This parameter will be setup by the toolstack (see alloc_magic_pages
> > in libxc/xc_dom_arm.c).
> > 
> > >been allocated by dom0. Whenever, an event is to be dispatched via
> > >evtchn_send() API in Xen, it can check if the event channel is the
> > >console event channel for a given domU. If yes and its source domain
> > >is dom0 and destination domain is domU then it will write the data
> > >back to the console out ring buffer of the domU and raise a console
> > >event to dom0.
> > >
> > >Once this interface is working, Xen HYP can check the source and
> > >destination dom ids and decide which way the event came from and
> > >accordingly process the console data. To allow a mix of PV console
> > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > >which tells whether Xen HYP should intercept and process the data (for
> > >pl011 UART case) or let it go transparently (for PV conosle case).
> > 
> > I am not very familiar with the event channel code. I will let the
> > others comment on this bit.
> > 
> > Regardless that, how would you decide whether the hypervisor should
> > intercept the notification?
> > 
> > I can see 2 different cases:
> > 	1) The guest is starting to use the pl011 then move to the HVC
> > console (or HVC then pl011)
> > 	2) The guest is using both the PL011 and the HVC console
> > 
> > Should we consider the second case valid? I would say yes, because a
> > user could specify both on the command line. If we use the same
> > ring, the output would be a total garbage.
> > 
> > So maybe we need to allocate two distinct rings and event channel?
> 
> This sounds like the only sensible thing to me.  I think this is really
> about adding a new device to the Xen virtual platform, and providing the
> user the option to choose which one he wants the tool in Dom0 to be
> presented using stdin/out. Presumably the other console/serial can be
> redirected to a file or socket or something?

Let me explain how the PV console protocol and drivers work, because
they are a bit unusual. The first PV console is advertised via
hvm_params. The guest calls:

  hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
  hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);

to get the two parameters to setup the ring and evtchn. If they are 0,
the guest considers the first console unavailable. Other PV console
rings, from the second onward, are advertised via xenstore like any
other Xen PV protocols. In those cases, frontend and backend access
xenstore to setup ring and event channel.

The PV console backends are unusual too. xenconsoled, available on all
Xen systems, is one process per host and can handle only one PV console
per domain. Specifically, it is only able to deal with the first console.
Domains that have multiple PV consoles require QEMU (not as an emulator,
but as a PV backends provider). The toolstack writes "type" =
"xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
pl011 PV consoles, but it wouldn't be the end of the world if we did.

Additionally, Xen cannot speak xenstore. It can neither read nor write
to it. I don't think we should add xenstore support to the hypervisor
for this. We need to come up with a solution that doesn't require it.

Finally, we cannot hijack one of the guest PV consoles, regardless of
whether it's the first console or one of the others, because the guest
can always try to use them at any time. We need a PV console reserved
for Xen-Dom0 communications on behalf of the guest. When a VM is created
with "pl011=y", the toolstack needs to allocate one more page and evtchn
for the exclusive hypervisor usage.  They are not going to be advertised
to the guest as PV consoles; otherwise, the guest could rightfully
access them.

Both Xen and the PV console backend need access to the two numbers (pfn
and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
If we use hvm_params for this, we need two new hvm_params and Xen needs
to unmap the pfn from the guest immediately, because we don't want the
guest to have access to it.

However, the PV console backend can access xenstore, so in that case, it
is fine to write the pfn and evtchn of the PV console for pl011 to
xenstore, paying attention at using the xenstore permissions
appropriately. There are no reasons why the guest should have access to
them; only the console backend should be able to read them. Given that
the console backend has dom0 privileges, it is not a problem. I also
suggest using new xenstore nodes, different from any of the existing PV
console nodes.  For example:

/local/domain/$DOMID/xen-console/$NUM/ring-ref
/local/domain/$DOMID/xen-console/$NUM/port

Where $DOMID is the guest domain id, and $NUM is the console number,
starting from 0. If we use new hvm_parms for the pl011 PV console, we
might get away without any xenstore stuff.

For simplicity, given that xenconsoled doesn't support multiple PV
consoles, we could setup the pl011 PV console *instead* of the regular
PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
safe if the toolstack doesn't provide a PV console. When pl011 is
requested, libxl could set the pfn and evtchn hvm_params to 0 for the
initial PV console. Eventually, it would be nice if xenconsoled was able
to support both consoles at the same time.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi Stefano,

Thanks for a detailed explanation. I have some queries.

>
> Let me explain how the PV console protocol and drivers work, because
> they are a bit unusual. The first PV console is advertised via
> hvm_params. The guest calls:
>
>   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
>   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
>
In drivers/tty/hvc/hvc_xen.c, I see that there are two console init
functions - xen_hvm_console_init () and xen_pv_console_init () and
either of them are called based on the xen_domain_type(). Can you
clarify which init () is called in which case?

> to get the two parameters to setup the ring and evtchn. If they are 0,
> the guest considers the first console unavailable. Other PV console
> rings, from the second onward, are advertised via xenstore like any
> other Xen PV protocols. In those cases, frontend and backend access
> xenstore to setup ring and event channel.
>
> The PV console backends are unusual too. xenconsoled, available on all
> Xen systems, is one process per host and can handle only one PV console
> per domain. Specifically, it is only able to deal with the first console.
> Domains that have multiple PV consoles require QEMU (not as an emulator,
> but as a PV backends provider). The toolstack writes "type" =
> "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> pl011 PV consoles, but it wouldn't be the end of the world if we did.
>
> Additionally, Xen cannot speak xenstore. It can neither read nor write
> to it. I don't think we should add xenstore support to the hypervisor
> for this. We need to come up with a solution that doesn't require it.
>
> Finally, we cannot hijack one of the guest PV consoles, regardless of
> whether it's the first console or one of the others, because the guest
> can always try to use them at any time. We need a PV console reserved

I see that that there are xen events which can be allocated
(alloc_unbound_xen_event_channel ()). Should I use those kind of
events or just allocate a new event for each VM (similar to how the
console event channel is allocated) and intercept and process them in
the Xen HYP?

> for Xen-Dom0 communications on behalf of the guest. When a VM is created
> with "pl011=y", the toolstack needs to allocate one more page and evtchn
> for the exclusive hypervisor usage.  They are not going to be advertised
> to the guest as PV consoles; otherwise, the guest could rightfully
> access them.
>
There is a function alloc_magic_pages () (defined in
tools/libxc/xc_dom_arm.c), where the console PFN and event channel are
allocated for a new domain. So can I allocate a new PFN and the event
channel here for pl011?

> Both Xen and the PV console backend need access to the two numbers (pfn
> and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> If we use hvm_params for this, we need two new hvm_params and Xen needs
> to unmap the pfn from the guest immediately, because we don't want the
> guest to have access to it.
>
This looks fine. We can add new hvm_params to communicate the PFN and
the event channel for Xen to use. These hvm_params can be called from
alloc_magic_pages().

> However, the PV console backend can access xenstore, so in that case, it
> is fine to write the pfn and evtchn of the PV console for pl011 to
> xenstore, paying attention at using the xenstore permissions
> appropriately. There are no reasons why the guest should have access to
> them; only the console backend should be able to read them. Given that
> the console backend has dom0 privileges, it is not a problem. I also
> suggest using new xenstore nodes, different from any of the existing PV
> console nodes.  For example:
>
> /local/domain/$DOMID/xen-console/$NUM/ring-ref
> /local/domain/$DOMID/xen-console/$NUM/port
>
> Where $DOMID is the guest domain id, and $NUM is the console number,
> starting from 0. If we use new hvm_parms for the pl011 PV console, we
> might get away without any xenstore stuff.
>
> For simplicity, given that xenconsoled doesn't support multiple PV
> consoles, we could setup the pl011 PV console *instead* of the regular
> PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> safe if the toolstack doesn't provide a PV console. When pl011 is
> requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> initial PV console. Eventually, it would be nice if xenconsoled was able
> to support both consoles at the same time.

Is domain_create_ring() responsible for setting up the ring and the
event channel for the first PV console? This function seems to refer
to xenstore for getting the shared page reference and the remote port.
Is this the same shared page which was set by libxl using hvm_param
for pfn (for guest to access).

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 20 Dec 2016, Bhupinder Thakur wrote:
> Hi Stefano,
> 
> Thanks for a detailed explanation. I have some queries.
> 
> > Let me explain how the PV console protocol and drivers work, because
> > they are a bit unusual. The first PV console is advertised via
> > hvm_params. The guest calls:
> >
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> >
> In drivers/tty/hvc/hvc_xen.c, I see that there are two console init
> functions - xen_hvm_console_init () and xen_pv_console_init () and
> either of them are called based on the xen_domain_type(). Can you
> clarify which init () is called in which case?

Good question. x86 Xen has two different types of guests, PV guests and
HVM guests. In Linux, many code paths depend on whether a guest is PV
or HVM. However, on ARM we only have one kind of guest which is neither
PV or HVM. In this case, initialization on ARM follows the HVM path, so
xen_hvm_console_init is the one that is called. It isn't always the case
though. Feel free to ask (or add a printk) if you are in doubt.


> > to get the two parameters to setup the ring and evtchn. If they are 0,
> > the guest considers the first console unavailable. Other PV console
> > rings, from the second onward, are advertised via xenstore like any
> > other Xen PV protocols. In those cases, frontend and backend access
> > xenstore to setup ring and event channel.
> >
> > The PV console backends are unusual too. xenconsoled, available on all
> > Xen systems, is one process per host and can handle only one PV console
> > per domain. Specifically, it is only able to deal with the first console.
> > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > but as a PV backends provider). The toolstack writes "type" =
> > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> >
> > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > to it. I don't think we should add xenstore support to the hypervisor
> > for this. We need to come up with a solution that doesn't require it.
> >
> > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > whether it's the first console or one of the others, because the guest
> > can always try to use them at any time. We need a PV console reserved
> 
> I see that that there are xen events which can be allocated
> (alloc_unbound_xen_event_channel ()). Should I use those kind of
> events or just allocate a new event for each VM (similar to how the
> console event channel is allocated) and intercept and process them in
> the Xen HYP?

You need to allocate a new event channel for the PV console. In fact, it
should be exactly like any other event channels for consoles.


> > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > for the exclusive hypervisor usage.  They are not going to be advertised
> > to the guest as PV consoles; otherwise, the guest could rightfully
> > access them.
> >
> There is a function alloc_magic_pages () (defined in
> tools/libxc/xc_dom_arm.c), where the console PFN and event channel are
> allocated for a new domain. So can I allocate a new PFN and the event
> channel here for pl011?

That is what I was thinking.


> > Both Xen and the PV console backend need access to the two numbers (pfn
> > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > to unmap the pfn from the guest immediately, because we don't want the
> > guest to have access to it.
> >
> This looks fine. We can add new hvm_params to communicate the PFN and
> the event channel for Xen to use. These hvm_params can be called from
> alloc_magic_pages().

That is the idea.


> > However, the PV console backend can access xenstore, so in that case, it
> > is fine to write the pfn and evtchn of the PV console for pl011 to
> > xenstore, paying attention at using the xenstore permissions
> > appropriately. There are no reasons why the guest should have access to
> > them; only the console backend should be able to read them. Given that
> > the console backend has dom0 privileges, it is not a problem. I also
> > suggest using new xenstore nodes, different from any of the existing PV
> > console nodes.  For example:
> >
> > /local/domain/$DOMID/xen-console/$NUM/ring-ref
> > /local/domain/$DOMID/xen-console/$NUM/port
> >
> > Where $DOMID is the guest domain id, and $NUM is the console number,
> > starting from 0. If we use new hvm_parms for the pl011 PV console, we
> > might get away without any xenstore stuff.
> >
> > For simplicity, given that xenconsoled doesn't support multiple PV
> > consoles, we could setup the pl011 PV console *instead* of the regular
> > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> > safe if the toolstack doesn't provide a PV console. When pl011 is
> > requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> > initial PV console. Eventually, it would be nice if xenconsoled was able
> > to support both consoles at the same time.
> 
> Is domain_create_ring() responsible for setting up the ring and the
> event channel for the first PV console?

tools/console/daemon/io.c:domain_create_ring is a xenconsoled function.
xenconsoled is the PV console backend. This function sets up the ring
from the backend point view, which means mapping the page and binding
the event channel. It doesn't do any allocations.


> This function seems to refer
> to xenstore for getting the shared page reference and the remote port.
> Is this the same shared page which was set by libxl using hvm_param
> for pfn (for guest to access).

That's right. It is the same.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi Stefano,

On 19/12/2016 21:24, Stefano Stabellini wrote:
> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>> (CC rest maintainers for event channel questions)
>>>
>>> On 16/12/16 10:06, Bhupinder Thakur wrote:
>>>> Hi,
>>>
>>> Hi Bhupinder,
>>>
>>>> The idea is for Xen to act as an intermediary as shown below:
>>>>
>>>>               ring buffers
>>>>                           rx/tx fifo
>>>> dom0 <-------------------> Xen HYP (running pl011 emulation)
>>>> <-------------------> domU
>>>>                    event
>>>>                           interrupts
>>>>
>>>> Xen will directly manage the in/out console ring buffers (allocated by
>>>> dom0 for dom0-domU console communication) for reading/writing console
>>>> data from/to dom0. On the other side, Xen HYP will emulate pl011 to
>>>> read/write data from/to domU and pass it on to/from dom0 over the
>>>> in/out console ring buffers. There should be no change in dom0 as it
>>>> will still use the same ring buffers. Similarly there should be no
>>>> change in domU which would be running a standard pll011 driver.
>>>>
>>>> Currently, I am working on the interface between dom0 and Xen HYP. I
>>>> want to intercept the console events in Xen HYP which pass between
>>>> dom0 and domU. For now, I just want to capture console data coming
>>> >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
>>>> interface is working.
>>>>
>>>> Since each guest domain will have a unique event channel assigned for
>>>> console communication, Xen HYP can find out the event channel for a
>>>> given domU from the start_info page of that domU, which should have
>>>
>>> The start_info page is x86 specific. If you want to get the console
>>> event channel for ARM, you would have to use
>>> d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
>>>
>>> This parameter will be setup by the toolstack (see alloc_magic_pages
>>> in libxc/xc_dom_arm.c).
>>>
>>>> been allocated by dom0. Whenever, an event is to be dispatched via
>>>> evtchn_send() API in Xen, it can check if the event channel is the
>>>> console event channel for a given domU. If yes and its source domain
>>>> is dom0 and destination domain is domU then it will write the data
>>>> back to the console out ring buffer of the domU and raise a console
>>>> event to dom0.
>>>>
>>>> Once this interface is working, Xen HYP can check the source and
>>>> destination dom ids and decide which way the event came from and
>>>> accordingly process the console data. To allow a mix of PV console
>>>> guests and pl011 guests, Xen might have to maintain a flag per domain,
>>>> which tells whether Xen HYP should intercept and process the data (for
>>>> pl011 UART case) or let it go transparently (for PV conosle case).
>>>
>>> I am not very familiar with the event channel code. I will let the
>>> others comment on this bit.
>>>
>>> Regardless that, how would you decide whether the hypervisor should
>>> intercept the notification?
>>>
>>> I can see 2 different cases:
>>> 	1) The guest is starting to use the pl011 then move to the HVC
>>> console (or HVC then pl011)
>>> 	2) The guest is using both the PL011 and the HVC console
>>>
>>> Should we consider the second case valid? I would say yes, because a
>>> user could specify both on the command line. If we use the same
>>> ring, the output would be a total garbage.
>>>
>>> So maybe we need to allocate two distinct rings and event channel?
>>
>> This sounds like the only sensible thing to me.  I think this is really
>> about adding a new device to the Xen virtual platform, and providing the
>> user the option to choose which one he wants the tool in Dom0 to be
>> presented using stdin/out. Presumably the other console/serial can be
>> redirected to a file or socket or something?
>
> Let me explain how the PV console protocol and drivers work, because
> they are a bit unusual. The first PV console is advertised via
> hvm_params. The guest calls:
>
>   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
>   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
>
> to get the two parameters to setup the ring and evtchn. If they are 0,
> the guest considers the first console unavailable. Other PV console
> rings, from the second onward, are advertised via xenstore like any
> other Xen PV protocols. In those cases, frontend and backend access
> xenstore to setup ring and event channel.
>
> The PV console backends are unusual too. xenconsoled, available on all
> Xen systems, is one process per host and can handle only one PV console
> per domain. Specifically, it is only able to deal with the first console.
> Domains that have multiple PV consoles require QEMU (not as an emulator,
> but as a PV backends provider). The toolstack writes "type" =
> "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> pl011 PV consoles, but it wouldn't be the end of the world if we did.
>
> Additionally, Xen cannot speak xenstore. It can neither read nor write
> to it. I don't think we should add xenstore support to the hypervisor
> for this. We need to come up with a solution that doesn't require it.

Agree on this.

>
> Finally, we cannot hijack one of the guest PV consoles, regardless of
> whether it's the first console or one of the others, because the guest
> can always try to use them at any time. We need a PV console reserved
> for Xen-Dom0 communications on behalf of the guest. When a VM is created
> with "pl011=y", the toolstack needs to allocate one more page and evtchn
> for the exclusive hypervisor usage.  They are not going to be advertised
> to the guest as PV consoles; otherwise, the guest could rightfully
> access them.
>
> Both Xen and the PV console backend need access to the two numbers (pfn
> and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> should use another way to tell pfn and evtchn to Xen, maybe hvm_params.

I think it will be the other way around. Xen will allocate the event 
channel and then report to the PV backends. Very similar to what it is 
done for ioreq server on x86 today.

> If we use hvm_params for this, we need two new hvm_params and Xen needs
> to unmap the pfn from the guest immediately, because we don't want the
> guest to have access to it.

If you unmap the pfn, the PV backend will not be able to request the 
page because there will be no translation available.

So what you want to do is preventing the guest to at least write into 
region (not sure if it is worth to restrict read) and unmap the page via 
the hypercall XENMEM_decrease_reservation.

>
> However, the PV console backend can access xenstore, so in that case, it
> is fine to write the pfn and evtchn of the PV console for pl011 to
> xenstore, paying attention at using the xenstore permissions
> appropriately. There are no reasons why the guest should have access to
> them; only the console backend should be able to read them. Given that
> the console backend has dom0 privileges, it is not a problem. I also
> suggest using new xenstore nodes, different from any of the existing PV
> console nodes.  For example:
>
> /local/domain/$DOMID/xen-console/$NUM/ring-ref
> /local/domain/$DOMID/xen-console/$NUM/port
>
> Where $DOMID is the guest domain id, and $NUM is the console number,
> starting from 0. If we use new hvm_parms for the pl011 PV console, we
> might get away without any xenstore stuff.
>
> For simplicity, given that xenconsoled doesn't support multiple PV
> consoles, we could setup the pl011 PV console *instead* of the regular
> PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> safe if the toolstack doesn't provide a PV console. When pl011 is
> requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> initial PV console. Eventually, it would be nice if xenconsoled was able
> to support both consoles at the same time.

The PL011 emulation will be slower than the PV console. While I think it 
is a sensible approach to either have PL011 or PV console, we will have 
to support both in the future.

IIRC, the UEFI firmware will use Xen console by default but I am not 
sure it will fallback to the PL011 if present. So we may require some 
change in the firmware to allow booting on different configuration (i.e 
PL011 guest or PV console guest).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 20 Dec 2016, Julien Grall wrote:
> Hi Stefano,
> 
> On 19/12/2016 21:24, Stefano Stabellini wrote:
> > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > (CC rest maintainers for event channel questions)
> > > > 
> > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > > Hi,
> > > > 
> > > > Hi Bhupinder,
> > > > 
> > > > > The idea is for Xen to act as an intermediary as shown below:
> > > > > 
> > > > >               ring buffers
> > > > >                           rx/tx fifo
> > > > > dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > > > <-------------------> domU
> > > > >                    event
> > > > >                           interrupts
> > > > > 
> > > > > Xen will directly manage the in/out console ring buffers (allocated by
> > > > > dom0 for dom0-domU console communication) for reading/writing console
> > > > > data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > > read/write data from/to domU and pass it on to/from dom0 over the
> > > > > in/out console ring buffers. There should be no change in dom0 as it
> > > > > will still use the same ring buffers. Similarly there should be no
> > > > > change in domU which would be running a standard pll011 driver.
> > > > > 
> > > > > Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > > want to intercept the console events in Xen HYP which pass between
> > > > > dom0 and domU. For now, I just want to capture console data coming
> > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > > interface is working.
> > > > > 
> > > > > Since each guest domain will have a unique event channel assigned for
> > > > > console communication, Xen HYP can find out the event channel for a
> > > > > given domU from the start_info page of that domU, which should have
> > > > 
> > > > The start_info page is x86 specific. If you want to get the console
> > > > event channel for ARM, you would have to use
> > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > 
> > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > in libxc/xc_dom_arm.c).
> > > > 
> > > > > been allocated by dom0. Whenever, an event is to be dispatched via
> > > > > evtchn_send() API in Xen, it can check if the event channel is the
> > > > > console event channel for a given domU. If yes and its source domain
> > > > > is dom0 and destination domain is domU then it will write the data
> > > > > back to the console out ring buffer of the domU and raise a console
> > > > > event to dom0.
> > > > > 
> > > > > Once this interface is working, Xen HYP can check the source and
> > > > > destination dom ids and decide which way the event came from and
> > > > > accordingly process the console data. To allow a mix of PV console
> > > > > guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > > > which tells whether Xen HYP should intercept and process the data (for
> > > > > pl011 UART case) or let it go transparently (for PV conosle case).
> > > > 
> > > > I am not very familiar with the event channel code. I will let the
> > > > others comment on this bit.
> > > > 
> > > > Regardless that, how would you decide whether the hypervisor should
> > > > intercept the notification?
> > > > 
> > > > I can see 2 different cases:
> > > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > > console (or HVC then pl011)
> > > > 	2) The guest is using both the PL011 and the HVC console
> > > > 
> > > > Should we consider the second case valid? I would say yes, because a
> > > > user could specify both on the command line. If we use the same
> > > > ring, the output would be a total garbage.
> > > > 
> > > > So maybe we need to allocate two distinct rings and event channel?
> > > 
> > > This sounds like the only sensible thing to me.  I think this is really
> > > about adding a new device to the Xen virtual platform, and providing the
> > > user the option to choose which one he wants the tool in Dom0 to be
> > > presented using stdin/out. Presumably the other console/serial can be
> > > redirected to a file or socket or something?
> > 
> > Let me explain how the PV console protocol and drivers work, because
> > they are a bit unusual. The first PV console is advertised via
> > hvm_params. The guest calls:
> > 
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> > 
> > to get the two parameters to setup the ring and evtchn. If they are 0,
> > the guest considers the first console unavailable. Other PV console
> > rings, from the second onward, are advertised via xenstore like any
> > other Xen PV protocols. In those cases, frontend and backend access
> > xenstore to setup ring and event channel.
> > 
> > The PV console backends are unusual too. xenconsoled, available on all
> > Xen systems, is one process per host and can handle only one PV console
> > per domain. Specifically, it is only able to deal with the first console.
> > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > but as a PV backends provider). The toolstack writes "type" =
> > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> > 
> > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > to it. I don't think we should add xenstore support to the hypervisor
> > for this. We need to come up with a solution that doesn't require it.
> 
> Agree on this.
> 
> > 
> > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > whether it's the first console or one of the others, because the guest
> > can always try to use them at any time. We need a PV console reserved
> > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > for the exclusive hypervisor usage.  They are not going to be advertised
> > to the guest as PV consoles; otherwise, the guest could rightfully
> > access them.
> > 
> > Both Xen and the PV console backend need access to the two numbers (pfn
> > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> 
> I think it will be the other way around. Xen will allocate the event channel
> and then report to the PV backends. Very similar to what it is done for ioreq
> server on x86 today.

It could work that way too. Even in the ioreq case though, the ioreq
parameters are still exposed via hvm_params (I am looking at
include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU).


> > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > to unmap the pfn from the guest immediately, because we don't want the
> > guest to have access to it.
> 
> If you unmap the pfn, the PV backend will not be able to request the page
> because there will be no translation available.
> 
> So what you want to do is preventing the guest to at least write into region
> (not sure if it is worth to restrict read) 

That's a good idea.


> and unmap the page via the hypercall XENMEM_decrease_reservation.

That would be issued by the guest itself, right? To save address space?


> > However, the PV console backend can access xenstore, so in that case, it
> > is fine to write the pfn and evtchn of the PV console for pl011 to
> > xenstore, paying attention at using the xenstore permissions
> > appropriately. There are no reasons why the guest should have access to
> > them; only the console backend should be able to read them. Given that
> > the console backend has dom0 privileges, it is not a problem. I also
> > suggest using new xenstore nodes, different from any of the existing PV
> > console nodes.  For example:
> > 
> > /local/domain/$DOMID/xen-console/$NUM/ring-ref
> > /local/domain/$DOMID/xen-console/$NUM/port
> > 
> > Where $DOMID is the guest domain id, and $NUM is the console number,
> > starting from 0. If we use new hvm_parms for the pl011 PV console, we
> > might get away without any xenstore stuff.
> > 
> > For simplicity, given that xenconsoled doesn't support multiple PV
> > consoles, we could setup the pl011 PV console *instead* of the regular
> > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> > safe if the toolstack doesn't provide a PV console. When pl011 is
> > requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> > initial PV console. Eventually, it would be nice if xenconsoled was able
> > to support both consoles at the same time.
> 
> The PL011 emulation will be slower than the PV console. While I think it is a
> sensible approach to either have PL011 or PV console, we will have to support
> both in the future.

It would be great if xenconsoled could support both simultaneously.


> IIRC, the UEFI firmware will use Xen console by default but I am not sure it
> will fallback to the PL011 if present. So we may require some change in the
> firmware to allow booting on different configuration (i.e PL011 guest or PV
> console guest).

Linux has checks to see if the pfn or evtchn are 0 and skips PV console
initialization in those cases. Tianocores probably has the same checks
(but I haven't read the code).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 20 Dec 2016, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Julien Grall wrote:
> > Hi Stefano,
> > 
> > On 19/12/2016 21:24, Stefano Stabellini wrote:
> > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > > (CC rest maintainers for event channel questions)
> > > > > 
> > > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > > > Hi,
> > > > > 
> > > > > Hi Bhupinder,
> > > > > 
> > > > > > The idea is for Xen to act as an intermediary as shown below:
> > > > > > 
> > > > > >               ring buffers
> > > > > >                           rx/tx fifo
> > > > > > dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > > > > <-------------------> domU
> > > > > >                    event
> > > > > >                           interrupts
> > > > > > 
> > > > > > Xen will directly manage the in/out console ring buffers (allocated by
> > > > > > dom0 for dom0-domU console communication) for reading/writing console
> > > > > > data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > > > read/write data from/to domU and pass it on to/from dom0 over the
> > > > > > in/out console ring buffers. There should be no change in dom0 as it
> > > > > > will still use the same ring buffers. Similarly there should be no
> > > > > > change in domU which would be running a standard pll011 driver.
> > > > > > 
> > > > > > Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > > > want to intercept the console events in Xen HYP which pass between
> > > > > > dom0 and domU. For now, I just want to capture console data coming
> > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > > > interface is working.
> > > > > > 
> > > > > > Since each guest domain will have a unique event channel assigned for
> > > > > > console communication, Xen HYP can find out the event channel for a
> > > > > > given domU from the start_info page of that domU, which should have
> > > > > 
> > > > > The start_info page is x86 specific. If you want to get the console
> > > > > event channel for ARM, you would have to use
> > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > > 
> > > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > > in libxc/xc_dom_arm.c).
> > > > > 
> > > > > > been allocated by dom0. Whenever, an event is to be dispatched via
> > > > > > evtchn_send() API in Xen, it can check if the event channel is the
> > > > > > console event channel for a given domU. If yes and its source domain
> > > > > > is dom0 and destination domain is domU then it will write the data
> > > > > > back to the console out ring buffer of the domU and raise a console
> > > > > > event to dom0.
> > > > > > 
> > > > > > Once this interface is working, Xen HYP can check the source and
> > > > > > destination dom ids and decide which way the event came from and
> > > > > > accordingly process the console data. To allow a mix of PV console
> > > > > > guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > > > > which tells whether Xen HYP should intercept and process the data (for
> > > > > > pl011 UART case) or let it go transparently (for PV conosle case).
> > > > > 
> > > > > I am not very familiar with the event channel code. I will let the
> > > > > others comment on this bit.
> > > > > 
> > > > > Regardless that, how would you decide whether the hypervisor should
> > > > > intercept the notification?
> > > > > 
> > > > > I can see 2 different cases:
> > > > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > > > console (or HVC then pl011)
> > > > > 	2) The guest is using both the PL011 and the HVC console
> > > > > 
> > > > > Should we consider the second case valid? I would say yes, because a
> > > > > user could specify both on the command line. If we use the same
> > > > > ring, the output would be a total garbage.
> > > > > 
> > > > > So maybe we need to allocate two distinct rings and event channel?
> > > > 
> > > > This sounds like the only sensible thing to me.  I think this is really
> > > > about adding a new device to the Xen virtual platform, and providing the
> > > > user the option to choose which one he wants the tool in Dom0 to be
> > > > presented using stdin/out. Presumably the other console/serial can be
> > > > redirected to a file or socket or something?
> > > 
> > > Let me explain how the PV console protocol and drivers work, because
> > > they are a bit unusual. The first PV console is advertised via
> > > hvm_params. The guest calls:
> > > 
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> > > 
> > > to get the two parameters to setup the ring and evtchn. If they are 0,
> > > the guest considers the first console unavailable. Other PV console
> > > rings, from the second onward, are advertised via xenstore like any
> > > other Xen PV protocols. In those cases, frontend and backend access
> > > xenstore to setup ring and event channel.
> > > 
> > > The PV console backends are unusual too. xenconsoled, available on all
> > > Xen systems, is one process per host and can handle only one PV console
> > > per domain. Specifically, it is only able to deal with the first console.
> > > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > > but as a PV backends provider). The toolstack writes "type" =
> > > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> > > 
> > > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > > to it. I don't think we should add xenstore support to the hypervisor
> > > for this. We need to come up with a solution that doesn't require it.
> > 
> > Agree on this.
> > 
> > > 
> > > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > > whether it's the first console or one of the others, because the guest
> > > can always try to use them at any time. We need a PV console reserved
> > > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > > for the exclusive hypervisor usage.  They are not going to be advertised
> > > to the guest as PV consoles; otherwise, the guest could rightfully
> > > access them.
> > > 
> > > Both Xen and the PV console backend need access to the two numbers (pfn
> > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > 
> > I think it will be the other way around. Xen will allocate the event channel
> > and then report to the PV backends. Very similar to what it is done for ioreq
> > server on x86 today.
> 
> It could work that way too. Even in the ioreq case though, the ioreq
> parameters are still exposed via hvm_params (I am looking at
> include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU).

Actually that's only the case for the default ioreq server (the first
ioreq, which is usually QEMU). Other ioreqs issue the
HVMOP_get_ioreq_server_info hypercall.

If we follow the same route, we would need to introduce something
similar for consoles. We would need a new hypercall.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi Stefano,

On 20/12/2016 20:53, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Julien Grall wrote:
>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>> Finally, we cannot hijack one of the guest PV consoles, regardless of
>>> whether it's the first console or one of the others, because the guest
>>> can always try to use them at any time. We need a PV console reserved
>>> for Xen-Dom0 communications on behalf of the guest. When a VM is created
>>> with "pl011=y", the toolstack needs to allocate one more page and evtchn
>>> for the exclusive hypervisor usage.  They are not going to be advertised
>>> to the guest as PV consoles; otherwise, the guest could rightfully
>>> access them.
>>>
>>> Both Xen and the PV console backend need access to the two numbers (pfn
>>> and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
>>> should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
>>
>> I think it will be the other way around. Xen will allocate the event channel
>> and then report to the PV backends. Very similar to what it is done for ioreq
>> server on x86 today.
>
> It could work that way too. Even in the ioreq case though, the ioreq
> parameters are still exposed via hvm_params (I am looking at
> include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU).

I am fine with exposing the event channel via hvm_params. My previous 
mail was more related of who is allocating the event channel.

 From my understanding, any event channel between Xen and a domain 
should currently be allocated by Xen via the function 
alloc_unbound_xen_event_channel.

>
>
>>> If we use hvm_params for this, we need two new hvm_params and Xen needs
>>> to unmap the pfn from the guest immediately, because we don't want the
>>> guest to have access to it.
>>
>> If you unmap the pfn, the PV backend will not be able to request the page
>> because there will be no translation available.
>>
>> So what you want to do is preventing the guest to at least write into region
>> (not sure if it is worth to restrict read)
>
> That's a good idea.
>
>
>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>
> That would be issued by the guest itself, right? To save address space?

Correct. The main use case today is ballooning, but guest could call it 
on any other RAM baked page.

I was thinking about more about the protection needed. Technically the 
data in the ring are not trusted. So if the guest is messing up with it, 
it would not be a big issue. Or did I miss anything here?

[...]

>> IIRC, the UEFI firmware will use Xen console by default but I am not sure it
>> will fallback to the PL011 if present. So we may require some change in the
>> firmware to allow booting on different configuration (i.e PL011 guest or PV
>> console guest).
>
> Linux has checks to see if the pfn or evtchn are 0 and skips PV console
> initialization in those cases. Tianocores probably has the same checks
> (but I haven't read the code).

Probably, I am more concerned about Tianocore console not falling back 
on the PL011. The UEFI firmware is specifically built for Xen ARM guest, 
so I would be surprised to see a PL011 driver included.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Wed, 21 Dec 2016, Julien Grall wrote:
> Hi Stefano,
> 
> On 20/12/2016 20:53, Stefano Stabellini wrote:
> > On Tue, 20 Dec 2016, Julien Grall wrote:
> > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > > > whether it's the first console or one of the others, because the guest
> > > > can always try to use them at any time. We need a PV console reserved
> > > > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > > > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > > > for the exclusive hypervisor usage.  They are not going to be advertised
> > > > to the guest as PV consoles; otherwise, the guest could rightfully
> > > > access them.
> > > > 
> > > > Both Xen and the PV console backend need access to the two numbers (pfn
> > > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > > 
> > > I think it will be the other way around. Xen will allocate the event
> > > channel
> > > and then report to the PV backends. Very similar to what it is done for
> > > ioreq
> > > server on x86 today.
> > 
> > It could work that way too. Even in the ioreq case though, the ioreq
> > parameters are still exposed via hvm_params (I am looking at
> > include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU).
> 
> I am fine with exposing the event channel via hvm_params. My previous mail was
> more related of who is allocating the event channel.
> 
> From my understanding, any event channel between Xen and a domain should
> currently be allocated by Xen via the function
> alloc_unbound_xen_event_channel.

OK
 
 
> > > > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > > > to unmap the pfn from the guest immediately, because we don't want the
> > > > guest to have access to it.
> > > 
> > > If you unmap the pfn, the PV backend will not be able to request the page
> > > because there will be no translation available.
> > > 
> > > So what you want to do is preventing the guest to at least write into
> > > region
> > > (not sure if it is worth to restrict read)
> > 
> > That's a good idea.
> > 
> > 
> > > and unmap the page via the hypercall XENMEM_decrease_reservation.
> > 
> > That would be issued by the guest itself, right? To save address space?
> 
> Correct. The main use case today is ballooning, but guest could call it on any
> other RAM baked page.
> 
> I was thinking about more about the protection needed. Technically the data in
> the ring are not trusted. So if the guest is messing up with it, it would not
> be a big issue. Or did I miss anything here?

I understand that a guest would be smart to call
XENMEM_decrease_reservation on the PV console page for pl011, but it
cannot be a security measure, because, in fact, it needs to be called by
the guest.  Of course, a malicious guest can simply not call
XENMEM_decrease_reservation for it.


> [...]
> 
> > > IIRC, the UEFI firmware will use Xen console by default but I am not sure
> > > it
> > > will fallback to the PL011 if present. So we may require some change in
> > > the
> > > firmware to allow booting on different configuration (i.e PL011 guest or
> > > PV
> > > console guest).
> > 
> > Linux has checks to see if the pfn or evtchn are 0 and skips PV console
> > initialization in those cases. Tianocores probably has the same checks
> > (but I haven't read the code).
> 
> Probably, I am more concerned about Tianocore console not falling back on the
> PL011. The UEFI firmware is specifically built for Xen ARM guest, so I would
> be surprised to see a PL011 driver included.

Good point.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi Stefano,

On 21/12/16 22:12, Stefano Stabellini wrote:
> On Wed, 21 Dec 2016, Julien Grall wrote:
>> On 20/12/2016 20:53, Stefano Stabellini wrote:
>>> On Tue, 20 Dec 2016, Julien Grall wrote:
>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>>>> If we use hvm_params for this, we need two new hvm_params and Xen needs
>>>>> to unmap the pfn from the guest immediately, because we don't want the
>>>>> guest to have access to it.
>>>>
>>>> If you unmap the pfn, the PV backend will not be able to request the page
>>>> because there will be no translation available.
>>>>
>>>> So what you want to do is preventing the guest to at least write into
>>>> region
>>>> (not sure if it is worth to restrict read)
>>>
>>> That's a good idea.
>>>
>>>
>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>>>
>>> That would be issued by the guest itself, right? To save address space?
>>
>> Correct. The main use case today is ballooning, but guest could call it on any
>> other RAM baked page.
>>
>> I was thinking about more about the protection needed. Technically the data in
>> the ring are not trusted. So if the guest is messing up with it, it would not
>> be a big issue. Or did I miss anything here?
>
> I understand that a guest would be smart to call
> XENMEM_decrease_reservation on the PV console page for pl011, but it
> cannot be a security measure, because, in fact, it needs to be called by
> the guest.  Of course, a malicious guest can simply not call
> XENMEM_decrease_reservation for it.

Sorry I was not clear. I was not suggested the guest to call 
XENMEM_decrease_reservation on ring for security but a malicious guest 
issuing the hypercall on the ring protected and replacing by another page.

This is the exact same problem as the one I mentioned on the ITS thread. 
The page live in guest memory but contains data that will only be 
touched by Xen.

If you remove those page from stage-2, the translation IPA -> MFN will 
be lost unless you store somewhere else. You would have to do it 
per-page as the buffer will use contiguous IPA but potentially 
noncontiguous MFN.

In the case of ITS the memory is provisioned by the guest. So there are 
not much to do there except adding protection in stage-2 such as write 
protection and preventing the guest to unmap it. However for the pl011 
ring, as Andrew pointed on IRC, what we need to do is accounting this 
page to the domain memory. No mapping is necessary in stage-2.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

On 28/12/16 17:49, Julien Grall wrote:
> On 21/12/16 22:12, Stefano Stabellini wrote:
>> On Wed, 21 Dec 2016, Julien Grall wrote:
>>> On 20/12/2016 20:53, Stefano Stabellini wrote:
>>>> On Tue, 20 Dec 2016, Julien Grall wrote:
>>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
>>>>>> needs
>>>>>> to unmap the pfn from the guest immediately, because we don't want
>>>>>> the
>>>>>> guest to have access to it.
>>>>>
>>>>> If you unmap the pfn, the PV backend will not be able to request
>>>>> the page
>>>>> because there will be no translation available.
>>>>>
>>>>> So what you want to do is preventing the guest to at least write into
>>>>> region
>>>>> (not sure if it is worth to restrict read)
>>>>
>>>> That's a good idea.
>>>>
>>>>
>>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>>>>
>>>> That would be issued by the guest itself, right? To save address space?
>>>
>>> Correct. The main use case today is ballooning, but guest could call
>>> it on any
>>> other RAM baked page.
>>>
>>> I was thinking about more about the protection needed. Technically
>>> the data in
>>> the ring are not trusted. So if the guest is messing up with it, it
>>> would not
>>> be a big issue. Or did I miss anything here?
>>
>> I understand that a guest would be smart to call
>> XENMEM_decrease_reservation on the PV console page for pl011, but it
>> cannot be a security measure, because, in fact, it needs to be called by
>> the guest.  Of course, a malicious guest can simply not call
>> XENMEM_decrease_reservation for it.
>
> Sorry I was not clear. I was not suggested the guest to call
> XENMEM_decrease_reservation on ring for security but a malicious guest
> issuing the hypercall on the ring protected and replacing by another page.
>
> This is the exact same problem as the one I mentioned on the ITS thread.
> The page live in guest memory but contains data that will only be
> touched by Xen.
>
> If you remove those page from stage-2, the translation IPA -> MFN will
> be lost unless you store somewhere else. You would have to do it
> per-page as the buffer will use contiguous IPA but potentially
> noncontiguous MFN.
>
> In the case of ITS the memory is provisioned by the guest. So there are
> not much to do there except adding protection in stage-2 such as write
> protection and preventing the guest to unmap it. However for the pl011
> ring, as Andrew pointed on IRC, what we need to do is accounting this
> page to the domain memory. No mapping is necessary in stage-2.

I forgot to mention. The that backend of PV console is already 
considering the data in the ring untrusted. So I think in this case, 
protection is not strictly necessary. The worst thing that could happen 
is the guest is corrupting one of its console. Not really a big deal.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> On 21/12/16 22:12, Stefano Stabellini wrote:
>>
>> On Wed, 21 Dec 2016, Julien Grall wrote:
>>>
>>> On 20/12/2016 20:53, Stefano Stabellini wrote:
>>>>
>>>> On Tue, 20 Dec 2016, Julien Grall wrote:
>>>>>
>>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>>>>>>
>>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>>>>>
>>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>>>>>
>>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
>>>>>> needs
>>>>>> to unmap the pfn from the guest immediately, because we don't want the
>>>>>> guest to have access to it.
>>>>>
>>>>>
>>>>> If you unmap the pfn, the PV backend will not be able to request the
>>>>> page
>>>>> because there will be no translation available.
>>>>>
>>>>> So what you want to do is preventing the guest to at least write into
>>>>> region
>>>>> (not sure if it is worth to restrict read)
>>>>
>>>>
>>>> That's a good idea.
>>>>
>>>>
>>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>>>>
>>>>
>>>> That would be issued by the guest itself, right? To save address space?
>>>
>>>
>>> Correct. The main use case today is ballooning, but guest could call it
>>> on any
>>> other RAM baked page.
>>>
>>> I was thinking about more about the protection needed. Technically the
>>> data in
>>> the ring are not trusted. So if the guest is messing up with it, it would
>>> not
>>> be a big issue. Or did I miss anything here?
>>
>>
>> I understand that a guest would be smart to call
>> XENMEM_decrease_reservation on the PV console page for pl011, but it
>> cannot be a security measure, because, in fact, it needs to be called by
>> the guest.  Of course, a malicious guest can simply not call
>> XENMEM_decrease_reservation for it.
>
>
> Sorry I was not clear. I was not suggested the guest to call
> XENMEM_decrease_reservation on ring for security but a malicious guest
> issuing the hypercall on the ring protected and replacing by another page.
>
> This is the exact same problem as the one I mentioned on the ITS thread. The
> page live in guest memory but contains data that will only be touched by
> Xen.
>
> If you remove those page from stage-2, the translation IPA -> MFN will be
> lost unless you store somewhere else. You would have to do it per-page as
> the buffer will use contiguous IPA but potentially noncontiguous MFN.
>
> In the case of ITS the memory is provisioned by the guest. So there are not
> much to do there except adding protection in stage-2 such as write
> protection and preventing the guest to unmap it. However for the pl011 ring,
> as Andrew pointed on IRC, what we need to do is accounting this page to the
> domain memory. No mapping is necessary in stage-2.

Please clarify what is meant by that no stage-2 mapping is required.
Does it mean that no stage-2 mapping is required for the guest as it
never needs to access this page?

However, the Xen HYP will need the stage-2 mapping to find out the
pl011 PFN --> physical MFN mapping so that it can map the page to its
own address space. Currently, I am using prepare_ring_for_helper () to
map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
VA.

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> > On 21/12/16 22:12, Stefano Stabellini wrote:
> >>
> >> On Wed, 21 Dec 2016, Julien Grall wrote:
> >>>
> >>> On 20/12/2016 20:53, Stefano Stabellini wrote:
> >>>>
> >>>> On Tue, 20 Dec 2016, Julien Grall wrote:
> >>>>>
> >>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
> >>>>>>
> >>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
> >>>>>>>
> >>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> >>>>>>
> >>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
> >>>>>> needs
> >>>>>> to unmap the pfn from the guest immediately, because we don't want the
> >>>>>> guest to have access to it.
> >>>>>
> >>>>>
> >>>>> If you unmap the pfn, the PV backend will not be able to request the
> >>>>> page
> >>>>> because there will be no translation available.
> >>>>>
> >>>>> So what you want to do is preventing the guest to at least write into
> >>>>> region
> >>>>> (not sure if it is worth to restrict read)
> >>>>
> >>>>
> >>>> That's a good idea.
> >>>>
> >>>>
> >>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
> >>>>
> >>>>
> >>>> That would be issued by the guest itself, right? To save address space?
> >>>
> >>>
> >>> Correct. The main use case today is ballooning, but guest could call it
> >>> on any
> >>> other RAM baked page.
> >>>
> >>> I was thinking about more about the protection needed. Technically the
> >>> data in
> >>> the ring are not trusted. So if the guest is messing up with it, it would
> >>> not
> >>> be a big issue. Or did I miss anything here?
> >>
> >>
> >> I understand that a guest would be smart to call
> >> XENMEM_decrease_reservation on the PV console page for pl011, but it
> >> cannot be a security measure, because, in fact, it needs to be called by
> >> the guest.  Of course, a malicious guest can simply not call
> >> XENMEM_decrease_reservation for it.
> >
> >
> > Sorry I was not clear. I was not suggested the guest to call
> > XENMEM_decrease_reservation on ring for security but a malicious guest
> > issuing the hypercall on the ring protected and replacing by another page.
> >
> > This is the exact same problem as the one I mentioned on the ITS thread. The
> > page live in guest memory but contains data that will only be touched by
> > Xen.
> >
> > If you remove those page from stage-2, the translation IPA -> MFN will be
> > lost unless you store somewhere else. You would have to do it per-page as
> > the buffer will use contiguous IPA but potentially noncontiguous MFN.
> >
> > In the case of ITS the memory is provisioned by the guest. So there are not
> > much to do there except adding protection in stage-2 such as write
> > protection and preventing the guest to unmap it. However for the pl011 ring,
> > as Andrew pointed on IRC, what we need to do is accounting this page to the
> > domain memory. No mapping is necessary in stage-2.
> 
> Please clarify what is meant by that no stage-2 mapping is required.
> Does it mean that no stage-2 mapping is required for the guest as it
> never needs to access this page?

That's right.


> However, the Xen HYP will need the stage-2 mapping to find out the
> pl011 PFN --> physical MFN mapping so that it can map the page to its
> own address space. Currently, I am using prepare_ring_for_helper () to
> map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
> VA.

I am not sure what Julien had in mind exactly. I like the idea of not
mapping the page at stage-2, but it is true that many interfaces expect
pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
knows the mfn and could use it to map the page, instead of the pfn.
However, the PV console backend also needs to map the same page, and it
currently does that by calling xc_map_foreign_range, which I believe
also expect a pfn.

So maybe it is easier to use write-protection in stage-2 (as for ITS),
unless Julien has a better idea?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,

I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to
the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for
a guest domain. I tried allocating a new SPI VIRQ by calling
vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests
is limited to 0 (vgic_num_irqs() return 32 and therefore in effect
says that no SPI VIRQ can be allocated because below 32 are for
PPI/SGI VIRQs).

I tried modifying the guest domU configuration to specify the SPI IRQs
to be allowed for the guest but that operation fails as it is not
allowed for the unprivileged guests. Is there a way to allow a SPI
VIRQ to be allocated to a guest OS. I could not find any instance in
the Xen where it is allocating a SPI VIRQ.

I was able to register a new PPI VIRQ for pl011 interrupt and could
inject that interrupt into the guest and handle the interrupt in the
guest.

Regards,
Bhupinder

On 4 January 2017 at 01:38, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
>> On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
>> > On 21/12/16 22:12, Stefano Stabellini wrote:
>> >>
>> >> On Wed, 21 Dec 2016, Julien Grall wrote:
>> >>>
>> >>> On 20/12/2016 20:53, Stefano Stabellini wrote:
>> >>>>
>> >>>> On Tue, 20 Dec 2016, Julien Grall wrote:
>> >>>>>
>> >>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>> >>>>>>
>> >>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>> >>>>>>>
>> >>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>> >>>>>>
>> >>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
>> >>>>>> needs
>> >>>>>> to unmap the pfn from the guest immediately, because we don't want the
>> >>>>>> guest to have access to it.
>> >>>>>
>> >>>>>
>> >>>>> If you unmap the pfn, the PV backend will not be able to request the
>> >>>>> page
>> >>>>> because there will be no translation available.
>> >>>>>
>> >>>>> So what you want to do is preventing the guest to at least write into
>> >>>>> region
>> >>>>> (not sure if it is worth to restrict read)
>> >>>>
>> >>>>
>> >>>> That's a good idea.
>> >>>>
>> >>>>
>> >>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>> >>>>
>> >>>>
>> >>>> That would be issued by the guest itself, right? To save address space?
>> >>>
>> >>>
>> >>> Correct. The main use case today is ballooning, but guest could call it
>> >>> on any
>> >>> other RAM baked page.
>> >>>
>> >>> I was thinking about more about the protection needed. Technically the
>> >>> data in
>> >>> the ring are not trusted. So if the guest is messing up with it, it would
>> >>> not
>> >>> be a big issue. Or did I miss anything here?
>> >>
>> >>
>> >> I understand that a guest would be smart to call
>> >> XENMEM_decrease_reservation on the PV console page for pl011, but it
>> >> cannot be a security measure, because, in fact, it needs to be called by
>> >> the guest.  Of course, a malicious guest can simply not call
>> >> XENMEM_decrease_reservation for it.
>> >
>> >
>> > Sorry I was not clear. I was not suggested the guest to call
>> > XENMEM_decrease_reservation on ring for security but a malicious guest
>> > issuing the hypercall on the ring protected and replacing by another page.
>> >
>> > This is the exact same problem as the one I mentioned on the ITS thread. The
>> > page live in guest memory but contains data that will only be touched by
>> > Xen.
>> >
>> > If you remove those page from stage-2, the translation IPA -> MFN will be
>> > lost unless you store somewhere else. You would have to do it per-page as
>> > the buffer will use contiguous IPA but potentially noncontiguous MFN.
>> >
>> > In the case of ITS the memory is provisioned by the guest. So there are not
>> > much to do there except adding protection in stage-2 such as write
>> > protection and preventing the guest to unmap it. However for the pl011 ring,
>> > as Andrew pointed on IRC, what we need to do is accounting this page to the
>> > domain memory. No mapping is necessary in stage-2.
>>
>> Please clarify what is meant by that no stage-2 mapping is required.
>> Does it mean that no stage-2 mapping is required for the guest as it
>> never needs to access this page?
>
> That's right.
>
>
>> However, the Xen HYP will need the stage-2 mapping to find out the
>> pl011 PFN --> physical MFN mapping so that it can map the page to its
>> own address space. Currently, I am using prepare_ring_for_helper () to
>> map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
>> VA.
>
> I am not sure what Julien had in mind exactly. I like the idea of not
> mapping the page at stage-2, but it is true that many interfaces expect
> pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> knows the mfn and could use it to map the page, instead of the pfn.
> However, the PV console backend also needs to map the same page, and it
> currently does that by calling xc_map_foreign_range, which I believe
> also expect a pfn.
>
> So maybe it is easier to use write-protection in stage-2 (as for ITS),
> unless Julien has a better idea?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

Hello Bhupinder,

it is good to hear that you are making progress.


On Fri, 6 Jan 2017, Bhupinder Thakur wrote:
> Hi,
> 
> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to
> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for
> a guest domain. I tried allocating a new SPI VIRQ by calling
> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests
> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect
> says that no SPI VIRQ can be allocated because below 32 are for
> PPI/SGI VIRQs).

Yes, you should use vgic_allocate_virq. For that to work, you need to
make sure that nr_spis for the domain is > 0. Given that nr_spis is
passed from libxl by calling xc_domain_create, I would set nr_spis to 1
in libxl__arch_domain_prepare_config.


> I tried modifying the guest domU configuration to specify the SPI IRQs
> to be allowed for the guest but that operation fails as it is not
> allowed for the unprivileged guests. Is there a way to allow a SPI
> VIRQ to be allocated to a guest OS. I could not find any instance in
> the Xen where it is allocating a SPI VIRQ.

If you have already tried what I suggested, could you please post the
error you are getting?


> I was able to register a new PPI VIRQ for pl011 interrupt and could
> inject that interrupt into the guest and handle the interrupt in the
> guest.
> 
> Regards,
> Bhupinder
> 
> On 4 January 2017 at 01:38, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> >> On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> >> > On 21/12/16 22:12, Stefano Stabellini wrote:
> >> >>
> >> >> On Wed, 21 Dec 2016, Julien Grall wrote:
> >> >>>
> >> >>> On 20/12/2016 20:53, Stefano Stabellini wrote:
> >> >>>>
> >> >>>> On Tue, 20 Dec 2016, Julien Grall wrote:
> >> >>>>>
> >> >>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
> >> >>>>>>
> >> >>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
> >> >>>>>>>
> >> >>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> >> >>>>>>
> >> >>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
> >> >>>>>> needs
> >> >>>>>> to unmap the pfn from the guest immediately, because we don't want the
> >> >>>>>> guest to have access to it.
> >> >>>>>
> >> >>>>>
> >> >>>>> If you unmap the pfn, the PV backend will not be able to request the
> >> >>>>> page
> >> >>>>> because there will be no translation available.
> >> >>>>>
> >> >>>>> So what you want to do is preventing the guest to at least write into
> >> >>>>> region
> >> >>>>> (not sure if it is worth to restrict read)
> >> >>>>
> >> >>>>
> >> >>>> That's a good idea.
> >> >>>>
> >> >>>>
> >> >>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
> >> >>>>
> >> >>>>
> >> >>>> That would be issued by the guest itself, right? To save address space?
> >> >>>
> >> >>>
> >> >>> Correct. The main use case today is ballooning, but guest could call it
> >> >>> on any
> >> >>> other RAM baked page.
> >> >>>
> >> >>> I was thinking about more about the protection needed. Technically the
> >> >>> data in
> >> >>> the ring are not trusted. So if the guest is messing up with it, it would
> >> >>> not
> >> >>> be a big issue. Or did I miss anything here?
> >> >>
> >> >>
> >> >> I understand that a guest would be smart to call
> >> >> XENMEM_decrease_reservation on the PV console page for pl011, but it
> >> >> cannot be a security measure, because, in fact, it needs to be called by
> >> >> the guest.  Of course, a malicious guest can simply not call
> >> >> XENMEM_decrease_reservation for it.
> >> >
> >> >
> >> > Sorry I was not clear. I was not suggested the guest to call
> >> > XENMEM_decrease_reservation on ring for security but a malicious guest
> >> > issuing the hypercall on the ring protected and replacing by another page.
> >> >
> >> > This is the exact same problem as the one I mentioned on the ITS thread. The
> >> > page live in guest memory but contains data that will only be touched by
> >> > Xen.
> >> >
> >> > If you remove those page from stage-2, the translation IPA -> MFN will be
> >> > lost unless you store somewhere else. You would have to do it per-page as
> >> > the buffer will use contiguous IPA but potentially noncontiguous MFN.
> >> >
> >> > In the case of ITS the memory is provisioned by the guest. So there are not
> >> > much to do there except adding protection in stage-2 such as write
> >> > protection and preventing the guest to unmap it. However for the pl011 ring,
> >> > as Andrew pointed on IRC, what we need to do is accounting this page to the
> >> > domain memory. No mapping is necessary in stage-2.
> >>
> >> Please clarify what is meant by that no stage-2 mapping is required.
> >> Does it mean that no stage-2 mapping is required for the guest as it
> >> never needs to access this page?
> >
> > That's right.
> >
> >
> >> However, the Xen HYP will need the stage-2 mapping to find out the
> >> pl011 PFN --> physical MFN mapping so that it can map the page to its
> >> own address space. Currently, I am using prepare_ring_for_helper () to
> >> map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
> >> VA.
> >
> > I am not sure what Julien had in mind exactly. I like the idea of not
> > mapping the page at stage-2, but it is true that many interfaces expect
> > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> > knows the mfn and could use it to map the page, instead of the pfn.
> > However, the PV console backend also needs to map the same page, and it
> > currently does that by calling xc_map_foreign_range, which I believe
> > also expect a pfn.
> >
> > So maybe it is easier to use write-protection in stage-2 (as for ITS),
> > unless Julien has a better idea?
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi Stefano,



On 7 January 2017 at 03:24, Stefano Stabellini <sstabellini@kernel.org> wrote:
> Hello Bhupinder,
>
> it is good to hear that you are making progress.
>
>
> On Fri, 6 Jan 2017, Bhupinder Thakur wrote:
>> Hi,
>>
>> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to
>> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for
>> a guest domain. I tried allocating a new SPI VIRQ by calling
>> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests
>> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect
>> says that no SPI VIRQ can be allocated because below 32 are for
>> PPI/SGI VIRQs).
>
> Yes, you should use vgic_allocate_virq. For that to work, you need to
> make sure that nr_spis for the domain is > 0. Given that nr_spis is
> passed from libxl by calling xc_domain_create, I would set nr_spis to 1
> in libxl__arch_domain_prepare_config.
>
I increment the nr_spis by 1 in domain_vgic_init() to ensure the
vgic_allocate_virq() is able to allocate a SPI for pl011. With that
change, I am able to inject a SPI irq into guest and handle it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 10 Jan 2017, Bhupinder Thakur wrote:
> Hi Stefano,
> 
> 
> 
> On 7 January 2017 at 03:24, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > Hello Bhupinder,
> >
> > it is good to hear that you are making progress.
> >
> >
> > On Fri, 6 Jan 2017, Bhupinder Thakur wrote:
> >> Hi,
> >>
> >> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to
> >> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for
> >> a guest domain. I tried allocating a new SPI VIRQ by calling
> >> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests
> >> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect
> >> says that no SPI VIRQ can be allocated because below 32 are for
> >> PPI/SGI VIRQs).
> >
> > Yes, you should use vgic_allocate_virq. For that to work, you need to
> > make sure that nr_spis for the domain is > 0. Given that nr_spis is
> > passed from libxl by calling xc_domain_create, I would set nr_spis to 1
> > in libxl__arch_domain_prepare_config.
> >
> I increment the nr_spis by 1 in domain_vgic_init() to ensure the
> vgic_allocate_virq() is able to allocate a SPI for pl011. With that
> change, I am able to inject a SPI irq into guest and handle it.

Great!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi,

On 06/01/17 21:54, Stefano Stabellini wrote:
> On Fri, 6 Jan 2017, Bhupinder Thakur wrote:
>> Hi,
>>
>> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to
>> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for
>> a guest domain. I tried allocating a new SPI VIRQ by calling
>> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests
>> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect
>> says that no SPI VIRQ can be allocated because below 32 are for
>> PPI/SGI VIRQs).
>
> Yes, you should use vgic_allocate_virq. For that to work, you need to
> make sure that nr_spis for the domain is > 0. Given that nr_spis is
> passed from libxl by calling xc_domain_create, I would set nr_spis to 1
> in libxl__arch_domain_prepare_config.

Before I forgot, this would need to be documented to allow people to 
easily use interrupt routing (for device passthrough) when PL011 enabled.

I would suggest to define the SPI used by the PL011 in 
include/public/arch-arm.h and the number of SPIs reserved 
(GUEST_NR_RESERVED_SPIS). This would avoid an hardcoded number in the 
code and easily extension whether we need more SPIs later on (I have the 
host bridge emulation in mind).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi,

Sorry for the late answer, I am just back from holidays and still 
catching-up with my e-mails.

On 03/01/17 20:08, Stefano Stabellini wrote:
> On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
>> On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
>>> On 21/12/16 22:12, Stefano Stabellini wrote:
>>>>
>>>> On Wed, 21 Dec 2016, Julien Grall wrote:
>>>>>
>>>>> On 20/12/2016 20:53, Stefano Stabellini wrote:
>>>>>>
>>>>>> On Tue, 20 Dec 2016, Julien Grall wrote:
>>>>>>>
>>>>>>> On 19/12/2016 21:24, Stefano Stabellini wrote:
>>>>>>>>
>>>>>>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>>>>>>>
>>>>>>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
>>>>>>>>
>>>>>>>> If we use hvm_params for this, we need two new hvm_params and Xen
>>>>>>>> needs
>>>>>>>> to unmap the pfn from the guest immediately, because we don't want the
>>>>>>>> guest to have access to it.
>>>>>>>
>>>>>>>
>>>>>>> If you unmap the pfn, the PV backend will not be able to request the
>>>>>>> page
>>>>>>> because there will be no translation available.
>>>>>>>
>>>>>>> So what you want to do is preventing the guest to at least write into
>>>>>>> region
>>>>>>> (not sure if it is worth to restrict read)
>>>>>>
>>>>>>
>>>>>> That's a good idea.
>>>>>>
>>>>>>
>>>>>>> and unmap the page via the hypercall XENMEM_decrease_reservation.
>>>>>>
>>>>>>
>>>>>> That would be issued by the guest itself, right? To save address space?
>>>>>
>>>>>
>>>>> Correct. The main use case today is ballooning, but guest could call it
>>>>> on any
>>>>> other RAM baked page.
>>>>>
>>>>> I was thinking about more about the protection needed. Technically the
>>>>> data in
>>>>> the ring are not trusted. So if the guest is messing up with it, it would
>>>>> not
>>>>> be a big issue. Or did I miss anything here?
>>>>
>>>>
>>>> I understand that a guest would be smart to call
>>>> XENMEM_decrease_reservation on the PV console page for pl011, but it
>>>> cannot be a security measure, because, in fact, it needs to be called by
>>>> the guest.  Of course, a malicious guest can simply not call
>>>> XENMEM_decrease_reservation for it.
>>>
>>>
>>> Sorry I was not clear. I was not suggested the guest to call
>>> XENMEM_decrease_reservation on ring for security but a malicious guest
>>> issuing the hypercall on the ring protected and replacing by another page.
>>>
>>> This is the exact same problem as the one I mentioned on the ITS thread. The
>>> page live in guest memory but contains data that will only be touched by
>>> Xen.
>>>
>>> If you remove those page from stage-2, the translation IPA -> MFN will be
>>> lost unless you store somewhere else. You would have to do it per-page as
>>> the buffer will use contiguous IPA but potentially noncontiguous MFN.
>>>
>>> In the case of ITS the memory is provisioned by the guest. So there are not
>>> much to do there except adding protection in stage-2 such as write
>>> protection and preventing the guest to unmap it. However for the pl011 ring,
>>> as Andrew pointed on IRC, what we need to do is accounting this page to the
>>> domain memory. No mapping is necessary in stage-2.
>>
>> Please clarify what is meant by that no stage-2 mapping is required.
>> Does it mean that no stage-2 mapping is required for the guest as it
>> never needs to access this page?
>
> That's right.
>
>
>> However, the Xen HYP will need the stage-2 mapping to find out the
>> pl011 PFN --> physical MFN mapping so that it can map the page to its
>> own address space. Currently, I am using prepare_ring_for_helper () to
>> map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
>> VA.
>
> I am not sure what Julien had in mind exactly. I like the idea of not
> mapping the page at stage-2, but it is true that many interfaces expect
> pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> knows the mfn and could use it to map the page, instead of the pfn.
> However, the PV console backend also needs to map the same page, and it
> currently does that by calling xc_map_foreign_range, which I believe
> also expect a pfn.

Do you agree that page such as ioreq and the pl011 PV console are only 
used for communication between Xen and a backend? We don't want the 
guest to access the content of the pages but still be able to map. I 
would guess that using the gfn was a convenience but has some security 
impact (see  guess using the gfn is mostly for convenience but has some 
security impact today (see XSA-197 and XSA-199). AFAIU, we also want to 
account those pages to a domain.

For the credit, this is a suggestion made by Andrew on IRC. If I 
remember correctly what he suggested, it would be to have an 
add-to-physmap version restricted for non-domU components (e.g backend, 
toolstack) which would allow to map a given page (maybe via an index?).

>
> So maybe it is easier to use write-protection in stage-2 (as for ITS),
> unless Julien has a better idea?
>

We need write-protection and also prevent the page to be removed.

But to be fair, I would not bother to try to write-protect it for now if 
we are going towards keeping the pl011 console in the guest memory. I 
don't think letting the guest writing in the console page is a big 
problem. The backend console should already be protected against 
malicious guest. If not, we are already in trouble as it part of the PV 
protocol :).

Regarding Xen, the console protocol is very easy and I think only bound 
checking is sufficient. So what a guest could do is hijacking its own 
console. Any thoughts?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 17 Jan 2017, Julien Grall wrote:
> Hi,
> 
> Sorry for the late answer, I am just back from holidays and still catching-up
> with my e-mails.
> 
> On 03/01/17 20:08, Stefano Stabellini wrote:
> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> > > On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> > > > On 21/12/16 22:12, Stefano Stabellini wrote:
> > > > > 
> > > > > On Wed, 21 Dec 2016, Julien Grall wrote:
> > > > > > 
> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote:
> > > > > > > 
> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote:
> > > > > > > > 
> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> > > > > > > > > 
> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > > > > > > > 
> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall
> > > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > If we use hvm_params for this, we need two new hvm_params and
> > > > > > > > > Xen
> > > > > > > > > needs
> > > > > > > > > to unmap the pfn from the guest immediately, because we don't
> > > > > > > > > want the
> > > > > > > > > guest to have access to it.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > If you unmap the pfn, the PV backend will not be able to request
> > > > > > > > the
> > > > > > > > page
> > > > > > > > because there will be no translation available.
> > > > > > > > 
> > > > > > > > So what you want to do is preventing the guest to at least write
> > > > > > > > into
> > > > > > > > region
> > > > > > > > (not sure if it is worth to restrict read)
> > > > > > > 
> > > > > > > 
> > > > > > > That's a good idea.
> > > > > > > 
> > > > > > > 
> > > > > > > > and unmap the page via the hypercall
> > > > > > > > XENMEM_decrease_reservation.
> > > > > > > 
> > > > > > > 
> > > > > > > That would be issued by the guest itself, right? To save address
> > > > > > > space?
> > > > > > 
> > > > > > 
> > > > > > Correct. The main use case today is ballooning, but guest could call
> > > > > > it
> > > > > > on any
> > > > > > other RAM baked page.
> > > > > > 
> > > > > > I was thinking about more about the protection needed. Technically
> > > > > > the
> > > > > > data in
> > > > > > the ring are not trusted. So if the guest is messing up with it, it
> > > > > > would
> > > > > > not
> > > > > > be a big issue. Or did I miss anything here?
> > > > > 
> > > > > 
> > > > > I understand that a guest would be smart to call
> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it
> > > > > cannot be a security measure, because, in fact, it needs to be called
> > > > > by
> > > > > the guest.  Of course, a malicious guest can simply not call
> > > > > XENMEM_decrease_reservation for it.
> > > > 
> > > > 
> > > > Sorry I was not clear. I was not suggested the guest to call
> > > > XENMEM_decrease_reservation on ring for security but a malicious guest
> > > > issuing the hypercall on the ring protected and replacing by another
> > > > page.
> > > > 
> > > > This is the exact same problem as the one I mentioned on the ITS thread.
> > > > The
> > > > page live in guest memory but contains data that will only be touched by
> > > > Xen.
> > > > 
> > > > If you remove those page from stage-2, the translation IPA -> MFN will
> > > > be
> > > > lost unless you store somewhere else. You would have to do it per-page
> > > > as
> > > > the buffer will use contiguous IPA but potentially noncontiguous MFN.
> > > > 
> > > > In the case of ITS the memory is provisioned by the guest. So there are
> > > > not
> > > > much to do there except adding protection in stage-2 such as write
> > > > protection and preventing the guest to unmap it. However for the pl011
> > > > ring,
> > > > as Andrew pointed on IRC, what we need to do is accounting this page to
> > > > the
> > > > domain memory. No mapping is necessary in stage-2.
> > > 
> > > Please clarify what is meant by that no stage-2 mapping is required.
> > > Does it mean that no stage-2 mapping is required for the guest as it
> > > never needs to access this page?
> > 
> > That's right.
> > 
> > 
> > > However, the Xen HYP will need the stage-2 mapping to find out the
> > > pl011 PFN --> physical MFN mapping so that it can map the page to its
> > > own address space. Currently, I am using prepare_ring_for_helper () to
> > > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
> > > VA.
> > 
> > I am not sure what Julien had in mind exactly. I like the idea of not
> > mapping the page at stage-2, but it is true that many interfaces expect
> > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> > knows the mfn and could use it to map the page, instead of the pfn.
> > However, the PV console backend also needs to map the same page, and it
> > currently does that by calling xc_map_foreign_range, which I believe
> > also expect a pfn.
> 
> Do you agree that page such as ioreq and the pl011 PV console are only used
> for communication between Xen and a backend? 

I agree


> We don't want the guest to access
> the content of the pages but still be able to map. I would guess that using
> the gfn was a convenience but has some security impact (see  guess using the
> gfn is mostly for convenience but has some security impact today (see XSA-197
> and XSA-199). AFAIU, we also want to account those pages to a domain.

Right


> For the credit, this is a suggestion made by Andrew on IRC. If I remember
> correctly what he suggested, it would be to have an add-to-physmap version
> restricted for non-domU components (e.g backend, toolstack) which would allow
> to map a given page (maybe via an index?).
>
> > 
> > So maybe it is easier to use write-protection in stage-2 (as for ITS),
> > unless Julien has a better idea?
> > 
> 
> We need write-protection and also prevent the page to be removed.
> 
> But to be fair, I would not bother to try to write-protect it for now if we
> are going towards keeping the pl011 console in the guest memory. I don't think
> letting the guest writing in the console page is a big problem. The backend
> console should already be protected against malicious guest. If not, we are
> already in trouble as it part of the PV protocol :).

True


> Regarding Xen, the console protocol is very easy and I think only bound
> checking is sufficient. So what a guest could do is hijacking its own console.
> Any thoughts?

I think you are correct, it is no different than providing a PV console
to a guest, which we already do.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,

I have done the changes for emulating pl011 in Xen. Currently, I have
verified the emulation code by manually reading/writing data to
/dev/ttyAMA0 which is the device file for pl011 device. The data is
flowing fine between xenconsoled and the guest domain.

As a next step, I wanted to use /dev/ttyAMA0 as a console.

For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
"extra" directive in the domU configuration file. However, I do not
see the output on the console once I attached the console using "xl
console <domain_name>". I tried using "xl console -t serial
<domain_name>" also but that shows the tty1 console and not the
ttyAMA0 one.

Note that currently for testing, I have patched the code in the
xenconsoled to read/write pl011 ring buffers and write them to the
same pty terminal as used by hvc0. Finally, once I have verified, I
will create a separate pty for pl011 and add a new console type
"pl011".

Regards,
Bhupinder


On 18 January 2017 at 00:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Tue, 17 Jan 2017, Julien Grall wrote:
>> Hi,
>>
>> Sorry for the late answer, I am just back from holidays and still catching-up
>> with my e-mails.
>>
>> On 03/01/17 20:08, Stefano Stabellini wrote:
>> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
>> > > On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
>> > > > On 21/12/16 22:12, Stefano Stabellini wrote:
>> > > > >
>> > > > > On Wed, 21 Dec 2016, Julien Grall wrote:
>> > > > > >
>> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote:
>> > > > > > >
>> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote:
>> > > > > > > >
>> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
>> > > > > > > > >
>> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall
>> > > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > If we use hvm_params for this, we need two new hvm_params and
>> > > > > > > > > Xen
>> > > > > > > > > needs
>> > > > > > > > > to unmap the pfn from the guest immediately, because we don't
>> > > > > > > > > want the
>> > > > > > > > > guest to have access to it.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > If you unmap the pfn, the PV backend will not be able to request
>> > > > > > > > the
>> > > > > > > > page
>> > > > > > > > because there will be no translation available.
>> > > > > > > >
>> > > > > > > > So what you want to do is preventing the guest to at least write
>> > > > > > > > into
>> > > > > > > > region
>> > > > > > > > (not sure if it is worth to restrict read)
>> > > > > > >
>> > > > > > >
>> > > > > > > That's a good idea.
>> > > > > > >
>> > > > > > >
>> > > > > > > > and unmap the page via the hypercall
>> > > > > > > > XENMEM_decrease_reservation.
>> > > > > > >
>> > > > > > >
>> > > > > > > That would be issued by the guest itself, right? To save address
>> > > > > > > space?
>> > > > > >
>> > > > > >
>> > > > > > Correct. The main use case today is ballooning, but guest could call
>> > > > > > it
>> > > > > > on any
>> > > > > > other RAM baked page.
>> > > > > >
>> > > > > > I was thinking about more about the protection needed. Technically
>> > > > > > the
>> > > > > > data in
>> > > > > > the ring are not trusted. So if the guest is messing up with it, it
>> > > > > > would
>> > > > > > not
>> > > > > > be a big issue. Or did I miss anything here?
>> > > > >
>> > > > >
>> > > > > I understand that a guest would be smart to call
>> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it
>> > > > > cannot be a security measure, because, in fact, it needs to be called
>> > > > > by
>> > > > > the guest.  Of course, a malicious guest can simply not call
>> > > > > XENMEM_decrease_reservation for it.
>> > > >
>> > > >
>> > > > Sorry I was not clear. I was not suggested the guest to call
>> > > > XENMEM_decrease_reservation on ring for security but a malicious guest
>> > > > issuing the hypercall on the ring protected and replacing by another
>> > > > page.
>> > > >
>> > > > This is the exact same problem as the one I mentioned on the ITS thread.
>> > > > The
>> > > > page live in guest memory but contains data that will only be touched by
>> > > > Xen.
>> > > >
>> > > > If you remove those page from stage-2, the translation IPA -> MFN will
>> > > > be
>> > > > lost unless you store somewhere else. You would have to do it per-page
>> > > > as
>> > > > the buffer will use contiguous IPA but potentially noncontiguous MFN.
>> > > >
>> > > > In the case of ITS the memory is provisioned by the guest. So there are
>> > > > not
>> > > > much to do there except adding protection in stage-2 such as write
>> > > > protection and preventing the guest to unmap it. However for the pl011
>> > > > ring,
>> > > > as Andrew pointed on IRC, what we need to do is accounting this page to
>> > > > the
>> > > > domain memory. No mapping is necessary in stage-2.
>> > >
>> > > Please clarify what is meant by that no stage-2 mapping is required.
>> > > Does it mean that no stage-2 mapping is required for the guest as it
>> > > never needs to access this page?
>> >
>> > That's right.
>> >
>> >
>> > > However, the Xen HYP will need the stage-2 mapping to find out the
>> > > pl011 PFN --> physical MFN mapping so that it can map the page to its
>> > > own address space. Currently, I am using prepare_ring_for_helper () to
>> > > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
>> > > VA.
>> >
>> > I am not sure what Julien had in mind exactly. I like the idea of not
>> > mapping the page at stage-2, but it is true that many interfaces expect
>> > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
>> > knows the mfn and could use it to map the page, instead of the pfn.
>> > However, the PV console backend also needs to map the same page, and it
>> > currently does that by calling xc_map_foreign_range, which I believe
>> > also expect a pfn.
>>
>> Do you agree that page such as ioreq and the pl011 PV console are only used
>> for communication between Xen and a backend?
>
> I agree
>
>
>> We don't want the guest to access
>> the content of the pages but still be able to map. I would guess that using
>> the gfn was a convenience but has some security impact (see  guess using the
>> gfn is mostly for convenience but has some security impact today (see XSA-197
>> and XSA-199). AFAIU, we also want to account those pages to a domain.
>
> Right
>
>
>> For the credit, this is a suggestion made by Andrew on IRC. If I remember
>> correctly what he suggested, it would be to have an add-to-physmap version
>> restricted for non-domU components (e.g backend, toolstack) which would allow
>> to map a given page (maybe via an index?).
>>
>> >
>> > So maybe it is easier to use write-protection in stage-2 (as for ITS),
>> > unless Julien has a better idea?
>> >
>>
>> We need write-protection and also prevent the page to be removed.
>>
>> But to be fair, I would not bother to try to write-protect it for now if we
>> are going towards keeping the pl011 console in the guest memory. I don't think
>> letting the guest writing in the console page is a big problem. The backend
>> console should already be protected against malicious guest. If not, we are
>> already in trouble as it part of the PV protocol :).
>
> True
>
>
>> Regarding Xen, the console protocol is very easy and I think only bound
>> checking is sufficient. So what a guest could do is hijacking its own console.
>> Any thoughts?
>
> I think you are correct, it is no different than providing a PV console
> to a guest, which we already do.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Fri, 27 Jan 2017, Bhupinder Thakur wrote:
> Hi,
> 
> I have done the changes for emulating pl011 in Xen. Currently, I have
> verified the emulation code by manually reading/writing data to
> /dev/ttyAMA0 which is the device file for pl011 device. The data is
> flowing fine between xenconsoled and the guest domain.

Good progress!


> As a next step, I wanted to use /dev/ttyAMA0 as a console.

Do you see, among the kernel boot messages, something about ttyAMA0
being available? Is there a getty -L ttyAMA0 in your inittab?


> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
> "extra" directive in the domU configuration file. However, I do not
> see the output on the console once I attached the console using "xl
> console <domain_name>". I tried using "xl console -t serial
> <domain_name>" also but that shows the tty1 console and not the
> ttyAMA0 one.
> 
> Note that currently for testing, I have patched the code in the
> xenconsoled to read/write pl011 ring buffers and write them to the
> same pty terminal as used by hvc0. Finally, once I have verified, I
> will create a separate pty for pl011 and add a new console type
> "pl011".

If you modified xenconsoled to use the same pty as hvc0, then you should
just do "xl console <domain_name>", the -t serial option is useless, as
there is only one console from xl console point of view.

You might want to disable the PV console for these tests. I suggest to
add a return at the beginning of drivers/tty/hvc/hvc_xen.c:xen_hvc_init
and drivers/tty/hvc/hvc_xen.c:xen_cons_init.



> Regards,
> Bhupinder
> 
> 
> On 18 January 2017 at 00:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Tue, 17 Jan 2017, Julien Grall wrote:
> >> Hi,
> >>
> >> Sorry for the late answer, I am just back from holidays and still catching-up
> >> with my e-mails.
> >>
> >> On 03/01/17 20:08, Stefano Stabellini wrote:
> >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> >> > > On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> >> > > > On 21/12/16 22:12, Stefano Stabellini wrote:
> >> > > > >
> >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote:
> >> > > > > >
> >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote:
> >> > > > > > >
> >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote:
> >> > > > > > > >
> >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> >> > > > > > > > >
> >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall
> >> > > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params and
> >> > > > > > > > > Xen
> >> > > > > > > > > needs
> >> > > > > > > > > to unmap the pfn from the guest immediately, because we don't
> >> > > > > > > > > want the
> >> > > > > > > > > guest to have access to it.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > If you unmap the pfn, the PV backend will not be able to request
> >> > > > > > > > the
> >> > > > > > > > page
> >> > > > > > > > because there will be no translation available.
> >> > > > > > > >
> >> > > > > > > > So what you want to do is preventing the guest to at least write
> >> > > > > > > > into
> >> > > > > > > > region
> >> > > > > > > > (not sure if it is worth to restrict read)
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > That's a good idea.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > > and unmap the page via the hypercall
> >> > > > > > > > XENMEM_decrease_reservation.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > That would be issued by the guest itself, right? To save address
> >> > > > > > > space?
> >> > > > > >
> >> > > > > >
> >> > > > > > Correct. The main use case today is ballooning, but guest could call
> >> > > > > > it
> >> > > > > > on any
> >> > > > > > other RAM baked page.
> >> > > > > >
> >> > > > > > I was thinking about more about the protection needed. Technically
> >> > > > > > the
> >> > > > > > data in
> >> > > > > > the ring are not trusted. So if the guest is messing up with it, it
> >> > > > > > would
> >> > > > > > not
> >> > > > > > be a big issue. Or did I miss anything here?
> >> > > > >
> >> > > > >
> >> > > > > I understand that a guest would be smart to call
> >> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it
> >> > > > > cannot be a security measure, because, in fact, it needs to be called
> >> > > > > by
> >> > > > > the guest.  Of course, a malicious guest can simply not call
> >> > > > > XENMEM_decrease_reservation for it.
> >> > > >
> >> > > >
> >> > > > Sorry I was not clear. I was not suggested the guest to call
> >> > > > XENMEM_decrease_reservation on ring for security but a malicious guest
> >> > > > issuing the hypercall on the ring protected and replacing by another
> >> > > > page.
> >> > > >
> >> > > > This is the exact same problem as the one I mentioned on the ITS thread.
> >> > > > The
> >> > > > page live in guest memory but contains data that will only be touched by
> >> > > > Xen.
> >> > > >
> >> > > > If you remove those page from stage-2, the translation IPA -> MFN will
> >> > > > be
> >> > > > lost unless you store somewhere else. You would have to do it per-page
> >> > > > as
> >> > > > the buffer will use contiguous IPA but potentially noncontiguous MFN.
> >> > > >
> >> > > > In the case of ITS the memory is provisioned by the guest. So there are
> >> > > > not
> >> > > > much to do there except adding protection in stage-2 such as write
> >> > > > protection and preventing the guest to unmap it. However for the pl011
> >> > > > ring,
> >> > > > as Andrew pointed on IRC, what we need to do is accounting this page to
> >> > > > the
> >> > > > domain memory. No mapping is necessary in stage-2.
> >> > >
> >> > > Please clarify what is meant by that no stage-2 mapping is required.
> >> > > Does it mean that no stage-2 mapping is required for the guest as it
> >> > > never needs to access this page?
> >> >
> >> > That's right.
> >> >
> >> >
> >> > > However, the Xen HYP will need the stage-2 mapping to find out the
> >> > > pl011 PFN --> physical MFN mapping so that it can map the page to its
> >> > > own address space. Currently, I am using prepare_ring_for_helper () to
> >> > > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
> >> > > VA.
> >> >
> >> > I am not sure what Julien had in mind exactly. I like the idea of not
> >> > mapping the page at stage-2, but it is true that many interfaces expect
> >> > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> >> > knows the mfn and could use it to map the page, instead of the pfn.
> >> > However, the PV console backend also needs to map the same page, and it
> >> > currently does that by calling xc_map_foreign_range, which I believe
> >> > also expect a pfn.
> >>
> >> Do you agree that page such as ioreq and the pl011 PV console are only used
> >> for communication between Xen and a backend?
> >
> > I agree
> >
> >
> >> We don't want the guest to access
> >> the content of the pages but still be able to map. I would guess that using
> >> the gfn was a convenience but has some security impact (see  guess using the
> >> gfn is mostly for convenience but has some security impact today (see XSA-197
> >> and XSA-199). AFAIU, we also want to account those pages to a domain.
> >
> > Right
> >
> >
> >> For the credit, this is a suggestion made by Andrew on IRC. If I remember
> >> correctly what he suggested, it would be to have an add-to-physmap version
> >> restricted for non-domU components (e.g backend, toolstack) which would allow
> >> to map a given page (maybe via an index?).
> >>
> >> >
> >> > So maybe it is easier to use write-protection in stage-2 (as for ITS),
> >> > unless Julien has a better idea?
> >> >
> >>
> >> We need write-protection and also prevent the page to be removed.
> >>
> >> But to be fair, I would not bother to try to write-protect it for now if we
> >> are going towards keeping the pl011 console in the guest memory. I don't think
> >> letting the guest writing in the console page is a big problem. The backend
> >> console should already be protected against malicious guest. If not, we are
> >> already in trouble as it part of the PV protocol :).
> >
> > True
> >
> >
> >> Regarding Xen, the console protocol is very easy and I think only bound
> >> checking is sufficient. So what a guest could do is hijacking its own console.
> >> Any thoughts?
> >
> > I think you are correct, it is no different than providing a PV console
> > to a guest, which we already do.
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Jan 27, 2017 at 05:44:15PM +0530, Bhupinder Thakur wrote:
> Hi,
> 
> I have done the changes for emulating pl011 in Xen. Currently, I have
> verified the emulation code by manually reading/writing data to
> /dev/ttyAMA0 which is the device file for pl011 device. The data is
> flowing fine between xenconsoled and the guest domain.
> 
> As a next step, I wanted to use /dev/ttyAMA0 as a console.
> 
> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
> "extra" directive in the domU configuration file. However, I do not
> see the output on the console once I attached the console using "xl
> console <domain_name>". I tried using "xl console -t serial
> <domain_name>" also but that shows the tty1 console and not the
> ttyAMA0 one.

I would try to verify that when Linux uses the pl011 as a console (as
opposed to just writing some characters into /dev/ttyAMA0), that it
doesn't error out somewhere.  So like Stefano suggests, make sure you
check the guest kernel log after boot (assuming you can SSH or something
into the box) and look at 'dmesg' to see if the pl011 driver failed.

You could also try adding some log info in you Xen pl011 emulation code
to see if the guest kernel driver prods the device in the sequence you
expect, based on looking at the linux driver for the pl011.

Thanks,
-Christoffer

> 
> 
> On 18 January 2017 at 00:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Tue, 17 Jan 2017, Julien Grall wrote:
> >> Hi,
> >>
> >> Sorry for the late answer, I am just back from holidays and still catching-up
> >> with my e-mails.
> >>
> >> On 03/01/17 20:08, Stefano Stabellini wrote:
> >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> >> > > On 28 December 2016 at 23:19, Julien Grall <julien.grall@arm.com> wrote:
> >> > > > On 21/12/16 22:12, Stefano Stabellini wrote:
> >> > > > >
> >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote:
> >> > > > > >
> >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote:
> >> > > > > > >
> >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote:
> >> > > > > > > >
> >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> >> > > > > > > > >
> >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall
> >> > > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params and
> >> > > > > > > > > Xen
> >> > > > > > > > > needs
> >> > > > > > > > > to unmap the pfn from the guest immediately, because we don't
> >> > > > > > > > > want the
> >> > > > > > > > > guest to have access to it.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > If you unmap the pfn, the PV backend will not be able to request
> >> > > > > > > > the
> >> > > > > > > > page
> >> > > > > > > > because there will be no translation available.
> >> > > > > > > >
> >> > > > > > > > So what you want to do is preventing the guest to at least write
> >> > > > > > > > into
> >> > > > > > > > region
> >> > > > > > > > (not sure if it is worth to restrict read)
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > That's a good idea.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > > and unmap the page via the hypercall
> >> > > > > > > > XENMEM_decrease_reservation.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > That would be issued by the guest itself, right? To save address
> >> > > > > > > space?
> >> > > > > >
> >> > > > > >
> >> > > > > > Correct. The main use case today is ballooning, but guest could call
> >> > > > > > it
> >> > > > > > on any
> >> > > > > > other RAM baked page.
> >> > > > > >
> >> > > > > > I was thinking about more about the protection needed. Technically
> >> > > > > > the
> >> > > > > > data in
> >> > > > > > the ring are not trusted. So if the guest is messing up with it, it
> >> > > > > > would
> >> > > > > > not
> >> > > > > > be a big issue. Or did I miss anything here?
> >> > > > >
> >> > > > >
> >> > > > > I understand that a guest would be smart to call
> >> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it
> >> > > > > cannot be a security measure, because, in fact, it needs to be called
> >> > > > > by
> >> > > > > the guest.  Of course, a malicious guest can simply not call
> >> > > > > XENMEM_decrease_reservation for it.
> >> > > >
> >> > > >
> >> > > > Sorry I was not clear. I was not suggested the guest to call
> >> > > > XENMEM_decrease_reservation on ring for security but a malicious guest
> >> > > > issuing the hypercall on the ring protected and replacing by another
> >> > > > page.
> >> > > >
> >> > > > This is the exact same problem as the one I mentioned on the ITS thread.
> >> > > > The
> >> > > > page live in guest memory but contains data that will only be touched by
> >> > > > Xen.
> >> > > >
> >> > > > If you remove those page from stage-2, the translation IPA -> MFN will
> >> > > > be
> >> > > > lost unless you store somewhere else. You would have to do it per-page
> >> > > > as
> >> > > > the buffer will use contiguous IPA but potentially noncontiguous MFN.
> >> > > >
> >> > > > In the case of ITS the memory is provisioned by the guest. So there are
> >> > > > not
> >> > > > much to do there except adding protection in stage-2 such as write
> >> > > > protection and preventing the guest to unmap it. However for the pl011
> >> > > > ring,
> >> > > > as Andrew pointed on IRC, what we need to do is accounting this page to
> >> > > > the
> >> > > > domain memory. No mapping is necessary in stage-2.
> >> > >
> >> > > Please clarify what is meant by that no stage-2 mapping is required.
> >> > > Does it mean that no stage-2 mapping is required for the guest as it
> >> > > never needs to access this page?
> >> >
> >> > That's right.
> >> >
> >> >
> >> > > However, the Xen HYP will need the stage-2 mapping to find out the
> >> > > pl011 PFN --> physical MFN mapping so that it can map the page to its
> >> > > own address space. Currently, I am using prepare_ring_for_helper () to
> >> > > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP
> >> > > VA.
> >> >
> >> > I am not sure what Julien had in mind exactly. I like the idea of not
> >> > mapping the page at stage-2, but it is true that many interfaces expect
> >> > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen
> >> > knows the mfn and could use it to map the page, instead of the pfn.
> >> > However, the PV console backend also needs to map the same page, and it
> >> > currently does that by calling xc_map_foreign_range, which I believe
> >> > also expect a pfn.
> >>
> >> Do you agree that page such as ioreq and the pl011 PV console are only used
> >> for communication between Xen and a backend?
> >
> > I agree
> >
> >
> >> We don't want the guest to access
> >> the content of the pages but still be able to map. I would guess that using
> >> the gfn was a convenience but has some security impact (see  guess using the
> >> gfn is mostly for convenience but has some security impact today (see XSA-197
> >> and XSA-199). AFAIU, we also want to account those pages to a domain.
> >
> > Right
> >
> >
> >> For the credit, this is a suggestion made by Andrew on IRC. If I remember
> >> correctly what he suggested, it would be to have an add-to-physmap version
> >> restricted for non-domU components (e.g backend, toolstack) which would allow
> >> to map a given page (maybe via an index?).
> >>
> >> >
> >> > So maybe it is easier to use write-protection in stage-2 (as for ITS),
> >> > unless Julien has a better idea?
> >> >
> >>
> >> We need write-protection and also prevent the page to be removed.
> >>
> >> But to be fair, I would not bother to try to write-protect it for now if we
> >> are going towards keeping the pl011 console in the guest memory. I don't think
> >> letting the guest writing in the console page is a big problem. The backend
> >> console should already be protected against malicious guest. If not, we are
> >> already in trouble as it part of the PV protocol :).
> >
> > True
> >
> >
> >> Regarding Xen, the console protocol is very easy and I think only bound
> >> checking is sufficient. So what a guest could do is hijacking its own console.
> >> Any thoughts?
> >
> > I think you are correct, it is no different than providing a PV console
> > to a guest, which we already do.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,

>> Hi,
>>
>> I have done the changes for emulating pl011 in Xen. Currently, I have
>> verified the emulation code by manually reading/writing data to
>> /dev/ttyAMA0 which is the device file for pl011 device. The data is
>> flowing fine between xenconsoled and the guest domain.
>>
>> As a next step, I wanted to use /dev/ttyAMA0 as a console.
>>
>> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
>> "extra" directive in the domU configuration file. However, I do not
>> see the output on the console once I attached the console using "xl
>> console <domain_name>". I tried using "xl console -t serial
>> <domain_name>" also but that shows the tty1 console and not the
>> ttyAMA0 one.
>
> I would try to verify that when Linux uses the pl011 as a console (as
> opposed to just writing some characters into /dev/ttyAMA0), that it
> doesn't error out somewhere.  So like Stefano suggests, make sure you
> check the guest kernel log after boot (assuming you can SSH or something
> into the box) and look at 'dmesg' to see if the pl011 driver failed.
>
> You could also try adding some log info in you Xen pl011 emulation code
> to see if the guest kernel driver prods the device in the sequence you
> expect, based on looking at the linux driver for the pl011.

I tested with the following changes:

1. Compiled out CONFIG_HVC_XEN
2. Specified "console=ttyAMA0" in the guest configuration

With some more changes/fixes, the guest console is working partially
over ttyAMA0. I can see the guest boot prints on the console. However,
the prints stop just towards the end when the init scripts are
executed.

So it seems that during boot, initially ttyAMA0 is used as a
write-only port, where the console driver keeps writing the output to
the port. Till this point, only tx has been enabled on ttyAMA0 (driver
probe() is called at this point). No irq/rx has been enabled yet. Just
as the init scripts are executed, the irq/rx is also enabled on
ttyAMA0 (driver startup() is called at this time which enables rx and
registers IRQ) and then the boot prints stop.

Another observation is that unless you compile out HVC_XEN support,
the hvc0 console also keeps receiving the data even if the
console=ttyAMA0.

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

On 03/02/17 13:53, Bhupinder Thakur wrote:
> Hi,

Hi Bhupinder,

>>> Hi,
>>>
>>> I have done the changes for emulating pl011 in Xen. Currently, I have
>>> verified the emulation code by manually reading/writing data to
>>> /dev/ttyAMA0 which is the device file for pl011 device. The data is
>>> flowing fine between xenconsoled and the guest domain.
>>>
>>> As a next step, I wanted to use /dev/ttyAMA0 as a console.
>>>
>>> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
>>> "extra" directive in the domU configuration file. However, I do not
>>> see the output on the console once I attached the console using "xl
>>> console <domain_name>". I tried using "xl console -t serial
>>> <domain_name>" also but that shows the tty1 console and not the
>>> ttyAMA0 one.
>>
>> I would try to verify that when Linux uses the pl011 as a console (as
>> opposed to just writing some characters into /dev/ttyAMA0), that it
>> doesn't error out somewhere.  So like Stefano suggests, make sure you
>> check the guest kernel log after boot (assuming you can SSH or something
>> into the box) and look at 'dmesg' to see if the pl011 driver failed.
>>
>> You could also try adding some log info in you Xen pl011 emulation code
>> to see if the guest kernel driver prods the device in the sequence you
>> expect, based on looking at the linux driver for the pl011.
>
> I tested with the following changes:
>
> 1. Compiled out CONFIG_HVC_XEN
> 2. Specified "console=ttyAMA0" in the guest configuration
>
> With some more changes/fixes, the guest console is working partially
> over ttyAMA0. I can see the guest boot prints on the console. However,
> the prints stop just towards the end when the init scripts are
> executed.
>
> So it seems that during boot, initially ttyAMA0 is used as a
> write-only port, where the console driver keeps writing the output to
> the port. Till this point, only tx has been enabled on ttyAMA0 (driver
> probe() is called at this point). No irq/rx has been enabled yet. Just
> as the init scripts are executed, the irq/rx is also enabled on
> ttyAMA0 (driver startup() is called at this time which enables rx and
> registers IRQ) and then the boot prints stop.

So if I understand correctly, you don't receive anymore output. Correct?
Have you tried to see whether the pl011 driver is receiving interrupt or 
even Linux calling it.

Assuming you have Xen built with debug enabled (see CONFIG_DEBUG), you 
could use xen_raw_printk in the pl011 driver to see what's going on. 
This will issue an hypercall and the message will be directly printed on 
Xen console.

If you compile out the HVC_XEN support, you could directly use 
HYPERVISOR_console_io(CONSOLEIO_write, len, str).

> Another observation is that unless you compile out HVC_XEN support,
> the hvc0 console also keeps receiving the data even if the
> console=ttyAMA0.

HVC is the preferred console when using Xen, I think you would also need 
to add "earlycon=pl0111,addr" on the kernel commandline where addr is 
the base address of the PL011 to get Linux using the serial port as boot 
console.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi Julien,

On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
> So if I understand correctly, you don't receive anymore output. Correct?
> Have you tried to see whether the pl011 driver is receiving interrupt or
> even Linux calling it.

The pl011 driver is not receiving the interrupts (I put a
HYPERVISOR_console_io() call inside the interrupt handler and I do not
see the print on the xen console) . When the console is in this state,
if I keep typing the characters on the console then Xen keeps raising
the interrupts (for each character typed) but all those interrupts are
marked as inflight. So it seems that interrupt is not getting cleared
in GIC. Which GIC function can I look at to further debug what is the
issue?

Earlier, when I used to manually write characters to /dev/ttyAMA0 then
I used to receive the interrupts all fine.

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
> Hi Julien,
> 
> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
> > So if I understand correctly, you don't receive anymore output. Correct?
> > Have you tried to see whether the pl011 driver is receiving interrupt or
> > even Linux calling it.
> 
> The pl011 driver is not receiving the interrupts (I put a
> HYPERVISOR_console_io() call inside the interrupt handler and I do not
> see the print on the xen console) . When the console is in this state,
> if I keep typing the characters on the console then Xen keeps raising
> the interrupts (for each character typed) but all those interrupts are
> marked as inflight. So it seems that interrupt is not getting cleared
> in GIC. Which GIC function can I look at to further debug what is the
> issue?
> 
> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
> I used to receive the interrupts all fine.

DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
mark the interrupt as a real hardware interrupt by mistake
(GICH_V2_LR_HW). At the same time, check that the virtual PL011
interrupt has been added correctly to the LRs
(xen/arch/arm/gic-v2.c:gicv2_update_lr).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

[-- Attachment #1: Type: text/plain, Size: 2985 bytes --]

Hi Stefano,

On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
>> Hi Julien,
>>
>> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
>> > So if I understand correctly, you don't receive anymore output. Correct?
>> > Have you tried to see whether the pl011 driver is receiving interrupt or
>> > even Linux calling it.
>>
>> The pl011 driver is not receiving the interrupts (I put a
>> HYPERVISOR_console_io() call inside the interrupt handler and I do not
>> see the print on the xen console) . When the console is in this state,
>> if I keep typing the characters on the console then Xen keeps raising
>> the interrupts (for each character typed) but all those interrupts are
>> marked as inflight. So it seems that interrupt is not getting cleared
>> in GIC. Which GIC function can I look at to further debug what is the
>> issue?
>>
>> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
>> I used to receive the interrupts all fine.
>
> DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
> mark the interrupt as a real hardware interrupt by mistake
> (GICH_V2_LR_HW). At the same time, check that the virtual PL011
> interrupt has been added correctly to the LRs
> (xen/arch/arm/gic-v2.c:gicv2_update_lr).

I verified that the PL011 interrupt is getting marked pending by Xen
but guest does not receive it. I am not sure if the local interrupt
delivery is disabled on the guest because of which it may not be
receiving the interrupts. Is there a way to verify that interrupts are
indeed disabled at that point. Since the interrupt delivery works fine
when I manually read/write characters to /dev/ttyAMA0, I think there
is no issue from the Xen side in terms of raising an interrupt.

One observation is that pl011_request_port () calls request_mem_region
() for the MMIO space allocated to pl011. The function returns -EBUSY
indicating that the memory region is already in use. But when I do cat
/proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
start of the pl011 MMIO space. I see the same error while booting the
dom0 kernel also.

I have attached the guest boot logs when console=ttyAMA0. I have added
debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
and serial_core.c. The output from the console seems to suddenly stop
once the pl011 port is started. I have not added getty for ttyAMA0 in
the /etc/inittab file. I believe auto-serial-console should be able to
attach a getty session to ttyAMA0 automatically (i did try with adding
a getty command in the inittab file but that did not solve the issue).

Towards the end of the log, we can see that every time a character is
typed, Xen raises an interrupt and the characters keep getting
collected in the IN RING buffer (in_prod keeps incrementing). Later
interrupts remain in inflight as the earlier interrupt is not cleared.

Regards,
Bhupinder

[-- Attachment #2: pl011.log --]
[-- Type: text/x-log, Size: 14134 bytes --]


(none):/root # cat domU.cfg
kernel = "/root/Image"
name = "guest"
memory = 512
vcpus = 1
extra = "console=ttyAMA0 root=/dev/xvda ro"
disk = [ 'phy:/dev/loop0,xvda,w' ]

(none):/root # xl create domU.cfg
Parsing config from domU.cfg

(XEN) allocated xen_consumer = 1 for port = 1, remote dom=0
(XEN) searching virq in < 32, 33 >
(XEN) HVM_PARAM_VPL011_VIRQ param get val=32
(XEN) HVM_PARAM_VPL011_CONSOLE_PFN = 0x39003
(XEN) HVM_PARAM_VPL011_CONSOLE_PFN param set val=0x39003
seq 1204 queued, 'add' 'xen-backend'

Validate module index
Check if link configuration needs reloading.
seq 1204 forked new worker [1556]
seq 1204 running
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15
IMPORT builtin 'hwdb' returned non-zero
RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Execute 'load' 'xen-backend:vbd'
passed device to netlink monitor 0xaaaab38dad90
seq 1204 processed
cleanup idle workers
Unload module index
Unloaded link configuration context.
worker [1556] exited
seq 1205 queued, 'add' 'xen-backend'
seq 1205 forked new worker [1614]
seq 1205 running
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15
IMPORT builtin 'hwdb' returned non-zero
RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Execute 'load' 'xen-backend:console'
passed device to netlink monitor 0xaaaab38dad90
seq 1205 processed
cleanup idle workers
Unload module index
Unloaded link configuration context.
worker [1614] exited


(none):/root # xl console guest

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.0-rc3-00243-g0c183d9-dirty (btuser@blr-ubuntu-linaro) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #94 SMP PREEMPT Mon Feb 13 12:12:48 IST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd000]
[    0.000000] Xen 4.9 support found
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 16 MiB at 0x000000005f000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] percpu: Embedded 21 pages/cpu @ffff80001efdc000 s48000 r8192 d29824 u86016
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129024
[    0.000000] Kernel command line: console=ttyAMA0 root=/dev/xvda ro
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] Memory: 476280K/524288K available (8380K kernel code, 876K rwdata, 3676K rodata, 1024K init, 283K bss, 31624K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff0000088b0000   (  8384 KB)
[    0.000000]     .rodata : 0xffff0000088b0000 - 0xffff000008c50000   (  3712 KB)
[    0.000000]       .init : 0xffff000008c50000 - 0xffff000008d50000   (  1024 KB)
[    0.000000]       .data : 0xffff000008d50000 - 0xffff000008e2b200   (   877 KB)
[    0.000000]        .bss : 0xffff000008e2b200 - 0xffff000008e71ec0   (   284 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e0000800000   (     8 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800020000000   (   512 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] Build-time adjustment of leaf fanout to 64.
[    0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 100.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000024] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000660] Console: colour dummy device 80x25
[    0.000720] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.000771] pid_max: default: 32768 minimum: 301
[    0.001138] Security Framework initialized
[    0.001338] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.001389] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.004333] ASID allocator initialised with 65536 entries
[    0.005545] xen:grant_table: Grant tables using version 1 layout
[    0.005653] Grant table initialized
[    0.005731] xen:events: Using FIFO-based ABI
[    0.005813] Xen: initializing cpu0
[    0.038093] EFI services will not be available.
[    0.052923] Brought up 1 CPUs
[    0.052955] SMP: Total of 1 processors activated.
[    0.052994] CPU features: detected feature: Privileged Access Never
[    0.053040] CPU features: detected feature: User Access Override
[    0.053094] CPU features: detected feature: 32-bit EL0 Support
[    0.053252] CPU: All CPU(s) started at EL1
[    0.053298] alternatives: patching kernel code
[    0.057685] devtmpfs: initialized
[    0.065647] DMI not present or invalid.
[    0.066549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.070126] pinctrl core: initialized pinctrl subsystem
[    0.075662] NET: Registered protocol family 16
[    0.090773] vdso: 2 pages (1 code @ ffff0000088b7000, 1 data @ ffff000008d54000)
[    0.090827] hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
[    0.096374] DMA: preallocated 256 KiB pool for atomic allocations
[    0.101159] Serial: AMBA PL011 UART driver
[    0.107941] [DEBUG] DT baudrate=115200
[    0.108053] [DEBUG] pl011_setup_port: mmiobase = 0x22000000
[    0.108082] [DEBUG] registering pl011 port...
[    0.108288] [DEBUG] pl011_request_port called with return -16
[    0.108321] [DEBUG] pl011_config_port return -16
[    0.108376] 22000000.sbsa-pl011: ttyAMA0 at MMIO 0x22000000 (irq = 5, base_baud = 0) is a SBSA
[    0.951889] console [ttyAMA0] enabled
[    1.079520] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.126153] ACPI: Interpreter disabled.
[    1.157902] xen:balloon: Initialising balloon driver
[    1.186169] xen_balloon: Initialising balloon driver
[    1.221471] vgaarb: loaded
[    1.253097] SCSI subsystem initialized
[    1.261315] usbcore: registered new interface driver usbfs
[    1.292045] usbcore: registered new interface driver hub
[    1.294535] usbcore: registered new device driver usb
[    1.299146] pps_core: LinuxPPS API ver. 1 registered
[    1.301146] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.305592] PTP clock support registered
[    1.309078] dmi: Firmware registration failed.
[    1.312554] Advanced Linux Sound Architecture Driver Initialized.
[    1.321390] clocksource: Switched to clocksource arch_sys_counter
[    1.351568] VFS: Disk quotas dquot_6.6.0
[    1.354063] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.357800] pnp: PnP ACPI: disabled
[    1.414296] NET: Registered protocol family 2
[    1.419180] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    1.423418] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
[    1.427725] TCP: Hash tables configured (established 4096 bind 4096)
[    1.431440] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    1.434356] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    1.437637] NET: Registered protocol family 1
[    1.448956] RPC: Registered named UNIX socket transport module.
[    1.479221] RPC: Registered udp transport module.
[    1.482021] RPC: Registered tcp transport module.
[    1.483433] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.487911] kvm [1]: HYP mode not available
[    1.492819] futex hash table entries: 256 (order: 3, 32768 bytes)
[    1.554422] audit: initializing netlink subsys (disabled)
[    1.556524] audit: type=2000 audit(0.708:1): initialized
[    1.565774] workingset: timestamp_bits=46 max_order=17 bucket_order=0
[    1.671737] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.680715] NFS: Registering the id_resolver key type
[    1.684352] Key type id_resolver registered
[    1.687312] Key type id_legacy registered
[    1.690059] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.693517] 9p: Installing v9fs 9p2000 file system support
[    1.711701] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.715750] io scheduler noop registered
[    1.719041] io scheduler cfq registered (default)
[    1.725609] libphy: mdio_driver_register: phy-bcm-ns2-pci
[    1.760436] xen:xen_evtchn: Event-channel device installed
[    1.791998] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.804472] SuperH (H)SCI(F) driver initialized
[    1.808884] msm_serial: driver initialized
[    1.812963] Unable to detect cache hierarchy from DT for CPU 0
[    1.870954] loop: module loaded
[    1.873610] Invalid max_queues (4), will use default max: 1.
[    1.885531] hisi_sas: driver version v1.6
[    1.947514] libphy: Fixed MDIO Bus: probed
[42080.693216] xen-blkback: backend/vbd/3/51712: using 1 queues, protocol 1 (arm-abi) persistent grants
[    2.004909] tun: Universal TUN/TAP device driver, 1.6
[    2.062463] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.122473] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    2.153762] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.157244] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    2.161758] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.165164] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    2.169878] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    2.173102] sky2: driver version 1.30
[    2.176530] QLogic/NetXen Network Driver v4.0.82
[    2.182603] xen_netfront: Initialising Xen virtual ethernet driver
[    2.193794] VFIO - User Level meta-driver version: 0.3
[    2.206433] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.234777] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.234789] ehci-pci: EHCI PCI platform driver
[    2.235235] ehci-platform: EHCI generic platform driver
[    2.235603] ehci-exynos: EHCI EXYNOS driver
[    2.235953] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    2.237978] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.238005] ohci-pci: OHCI PCI platform driver
[    2.238323] ohci-platform: OHCI generic platform driver
[    2.238667] ohci-exynos: OHCI EXYNOS driver
[    2.240473] usbcore: registered new interface driver usb-storage
[    2.247037] mousedev: PS/2 mouse device common for all mice
[    2.254747] i2c /dev entries driver
[    2.264399] xen_wdt: Xen WatchDog Timer Driver v0.01
[    2.267832] xen_wdt: initialized (timeout=60s, nowayout=0)
[    2.269948] sdhci: Secure Digital Host Controller Interface driver
[    2.269957] sdhci: Copyright(c) Pierre Ossman
[    2.271099] Synopsys Designware Multimedia Card Interface Driver
[    2.273296] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.276918] ledtrig-cpu: registered to indicate activity on CPUs
[    2.284412] usbcore: registered new interface driver usbhid
[    2.284420] usbhid: USB HID core driver
[    2.296057] NET: Registered protocol family 17
[    2.296431] 9pnet: Installing 9P2000 support
[    2.297058] Key type dns_resolver registered
[    2.305379] registered taskstats version 1
[    2.409868] hctosys: unable to open rtc device (rtc0)
[    2.412447] ALSA device list:
[    2.414132]   No soundcards found.
[    2.415352] [DEBUG] uart_port_startup entry with port type = 32
[    2.418303] [DEBUG] sbsa_uart_startup called...
[    2.419736] [DEBUG] pl011_allocate_irq irq=5...
[    2.421320] [DEBUG] sbsa_uart_startup done...
[    2.568210] [DEBUG] uart_port_startup exit
[    2.570649] EXT4-fs (xvda): mounting ext3 file system using the ext4 subsystem
[    2.615662] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
[    2.644906] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
[    2.653947] devtmpfs: mounted
[    2.685517] Freeing unused kernel memory: 1024K (ffff800000c50000 - ffff800000d50000)
[    2.750920] [DEBUG] uart_wait_until_sent entry ...
[    2.780537] [DEBUG] pl011_tx_empty called with return 1
[    2.781983] [DEBUG] uart_wait_until_sent exit ...
[    2.783592] [DEBUG] uart_wait_until_sent entry ...
[    2.786313] [DEBUG] pl011_tx_empty called with return 1
[    2.787676] [DEBUG] uart_wait_until_sent exit ...
(XEN) [DEBUG]SENding guest pl011 intr 32 with intr=0x30, flag=0x80, out_cons=12494, out_prod=12494, in_cons=0, in_prod=1, status=0...
(XEN) [DEBUG]vgic_vcpu_inject_irq: raised
(XEN) [DEBUG]gic_raise_guest_irq: pending
(XEN) [DEBUG]irq 32 marked lr pending
(XEN) [DEBUG]setting lr 1 for irq 32
(XEN) [DEBUG]gicv2_update_lr: 1, 1
(XEN) [DEBUG]SENding guest pl011 intr 32 with intr=0x30, flag=0x80, out_cons=12494, out_prod=12494, in_cons=0, in_prod=2, status=0...
(XEN) [DEBUG]vgic_vcpu_inject_irq: inflight

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Mon, 13 Feb 2017, Bhupinder Thakur wrote:
> Hi Stefano,
> 
> On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
> >> Hi Julien,
> >>
> >> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
> >> > So if I understand correctly, you don't receive anymore output. Correct?
> >> > Have you tried to see whether the pl011 driver is receiving interrupt or
> >> > even Linux calling it.
> >>
> >> The pl011 driver is not receiving the interrupts (I put a
> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not
> >> see the print on the xen console) . When the console is in this state,
> >> if I keep typing the characters on the console then Xen keeps raising
> >> the interrupts (for each character typed) but all those interrupts are
> >> marked as inflight. So it seems that interrupt is not getting cleared
> >> in GIC. Which GIC function can I look at to further debug what is the
> >> issue?
> >>
> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
> >> I used to receive the interrupts all fine.
> >
> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
> > mark the interrupt as a real hardware interrupt by mistake
> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011
> > interrupt has been added correctly to the LRs
> > (xen/arch/arm/gic-v2.c:gicv2_update_lr).
> 
> I verified that the PL011 interrupt is getting marked pending by Xen
> but guest does not receive it. I am not sure if the local interrupt
> delivery is disabled on the guest because of which it may not be
> receiving the interrupts. Is there a way to verify that interrupts are
> indeed disabled at that point. Since the interrupt delivery works fine
> when I manually read/write characters to /dev/ttyAMA0, I think there
> is no issue from the Xen side in terms of raising an interrupt.
> 
> One observation is that pl011_request_port () calls request_mem_region
> () for the MMIO space allocated to pl011. The function returns -EBUSY
> indicating that the memory region is already in use. But when I do cat
> /proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
> start of the pl011 MMIO space. I see the same error while booting the
> dom0 kernel also.
> 
> I have attached the guest boot logs when console=ttyAMA0. I have added
> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
> and serial_core.c. The output from the console seems to suddenly stop
> once the pl011 port is started. I have not added getty for ttyAMA0 in
> the /etc/inittab file. I believe auto-serial-console should be able to
> attach a getty session to ttyAMA0 automatically (i did try with adding
> a getty command in the inittab file but that did not solve the issue).
> 
> Towards the end of the log, we can see that every time a character is
> typed, Xen raises an interrupt and the characters keep getting
> collected in the IN RING buffer (in_prod keeps incrementing). Later
> interrupts remain in inflight as the earlier interrupt is not cleared.

Let me get this straight: who is printing all those messages we see on
the guest console (xl console guest) up until "Freeing unused kernel
memory"?  Is it the early pl011 driver? Or is it the Xen PV console
driver?

Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to
issue HYPERVISOR_console_io hypercalls. You might want to check that
do_console_io in Xen is letting DomU calls through.

Otherwise pl011_int might actually be called but the printk might not
show up on the Xen console.

The interrupt is enabled from Xen point of view, otherwise it would not
be added to the LR. However Linux might still refuse to call the
interrupt handler for some reason. You might want to add a printk in the
Linux generic interrupt handler, probably
kernel/irq/chip.c:handle_fasteoi_irq in your case.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi Stefano,

On 14 February 2017 at 03:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Mon, 13 Feb 2017, Bhupinder Thakur wrote:
>> Hi Stefano,
>>
>> On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
>> >> Hi Julien,
>> >>
>> >> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
>> >> > So if I understand correctly, you don't receive anymore output. Correct?
>> >> > Have you tried to see whether the pl011 driver is receiving interrupt or
>> >> > even Linux calling it.
>> >>
>> >> The pl011 driver is not receiving the interrupts (I put a
>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not
>> >> see the print on the xen console) . When the console is in this state,
>> >> if I keep typing the characters on the console then Xen keeps raising
>> >> the interrupts (for each character typed) but all those interrupts are
>> >> marked as inflight. So it seems that interrupt is not getting cleared
>> >> in GIC. Which GIC function can I look at to further debug what is the
>> >> issue?
>> >>
>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
>> >> I used to receive the interrupts all fine.
>> >
>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
>> > mark the interrupt as a real hardware interrupt by mistake
>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011
>> > interrupt has been added correctly to the LRs
>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr).
>>
>> I verified that the PL011 interrupt is getting marked pending by Xen
>> but guest does not receive it. I am not sure if the local interrupt
>> delivery is disabled on the guest because of which it may not be
>> receiving the interrupts. Is there a way to verify that interrupts are
>> indeed disabled at that point. Since the interrupt delivery works fine
>> when I manually read/write characters to /dev/ttyAMA0, I think there
>> is no issue from the Xen side in terms of raising an interrupt.
>>
>> One observation is that pl011_request_port () calls request_mem_region
>> () for the MMIO space allocated to pl011. The function returns -EBUSY
>> indicating that the memory region is already in use. But when I do cat
>> /proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
>> start of the pl011 MMIO space. I see the same error while booting the
>> dom0 kernel also.
>>
>> I have attached the guest boot logs when console=ttyAMA0. I have added
>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
>> and serial_core.c. The output from the console seems to suddenly stop
>> once the pl011 port is started. I have not added getty for ttyAMA0 in
>> the /etc/inittab file. I believe auto-serial-console should be able to
>> attach a getty session to ttyAMA0 automatically (i did try with adding
>> a getty command in the inittab file but that did not solve the issue).
>>
>> Towards the end of the log, we can see that every time a character is
>> typed, Xen raises an interrupt and the characters keep getting
>> collected in the IN RING buffer (in_prod keeps incrementing). Later
>> interrupts remain in inflight as the earlier interrupt is not cleared.
>
> Let me get this straight: who is printing all those messages we see on
> the guest console (xl console guest) up until "Freeing unused kernel
> memory"?  Is it the early pl011 driver? Or is it the Xen PV console
> driver?
>
> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to
> issue HYPERVISOR_console_io hypercalls. You might want to check that
> do_console_io in Xen is letting DomU calls through.
>
> Otherwise pl011_int might actually be called but the printk might not
> show up on the Xen console.
>
> The interrupt is enabled from Xen point of view, otherwise it would not
> be added to the LR. However Linux might still refuse to call the
> interrupt handler for some reason. You might want to add a printk in the
> Linux generic interrupt handler, probably
> kernel/irq/chip.c:handle_fasteoi_irq in your case.

The pl011 console is working now. I did not fix any thing specific
except I cleaned up the debug code which I had added. I am not sure if
some of my debug code was causing the issue.

After cleaning up the debug code and building the code afresh, I
started receiving the interrupts and input/output on the console is
working fine. Now I am able to execute commands on the console.

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,



On 15 February 2017 at 13:45, Bhupinder Thakur
<bhupinder.thakur@linaro.org> wrote:
> Hi Stefano,
>
> On 14 February 2017 at 03:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> On Mon, 13 Feb 2017, Bhupinder Thakur wrote:
>>> Hi Stefano,
>>>
>>> On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
>>> >> Hi Julien,
>>> >>
>>> >> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
>>> >> > So if I understand correctly, you don't receive anymore output. Correct?
>>> >> > Have you tried to see whether the pl011 driver is receiving interrupt or
>>> >> > even Linux calling it.
>>> >>
>>> >> The pl011 driver is not receiving the interrupts (I put a
>>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not
>>> >> see the print on the xen console) . When the console is in this state,
>>> >> if I keep typing the characters on the console then Xen keeps raising
>>> >> the interrupts (for each character typed) but all those interrupts are
>>> >> marked as inflight. So it seems that interrupt is not getting cleared
>>> >> in GIC. Which GIC function can I look at to further debug what is the
>>> >> issue?
>>> >>
>>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
>>> >> I used to receive the interrupts all fine.
>>> >
>>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
>>> > mark the interrupt as a real hardware interrupt by mistake
>>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011
>>> > interrupt has been added correctly to the LRs
>>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr).
>>>
>>> I verified that the PL011 interrupt is getting marked pending by Xen
>>> but guest does not receive it. I am not sure if the local interrupt
>>> delivery is disabled on the guest because of which it may not be
>>> receiving the interrupts. Is there a way to verify that interrupts are
>>> indeed disabled at that point. Since the interrupt delivery works fine
>>> when I manually read/write characters to /dev/ttyAMA0, I think there
>>> is no issue from the Xen side in terms of raising an interrupt.
>>>
>>> One observation is that pl011_request_port () calls request_mem_region
>>> () for the MMIO space allocated to pl011. The function returns -EBUSY
>>> indicating that the memory region is already in use. But when I do cat
>>> /proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
>>> start of the pl011 MMIO space. I see the same error while booting the
>>> dom0 kernel also.
>>>
>>> I have attached the guest boot logs when console=ttyAMA0. I have added
>>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
>>> and serial_core.c. The output from the console seems to suddenly stop
>>> once the pl011 port is started. I have not added getty for ttyAMA0 in
>>> the /etc/inittab file. I believe auto-serial-console should be able to
>>> attach a getty session to ttyAMA0 automatically (i did try with adding
>>> a getty command in the inittab file but that did not solve the issue).
>>>
>>> Towards the end of the log, we can see that every time a character is
>>> typed, Xen raises an interrupt and the characters keep getting
>>> collected in the IN RING buffer (in_prod keeps incrementing). Later
>>> interrupts remain in inflight as the earlier interrupt is not cleared.
>>
>> Let me get this straight: who is printing all those messages we see on
>> the guest console (xl console guest) up until "Freeing unused kernel
>> memory"?  Is it the early pl011 driver? Or is it the Xen PV console
>> driver?
>>
>> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to
>> issue HYPERVISOR_console_io hypercalls. You might want to check that
>> do_console_io in Xen is letting DomU calls through.
>>
>> Otherwise pl011_int might actually be called but the printk might not
>> show up on the Xen console.
>>
>> The interrupt is enabled from Xen point of view, otherwise it would not
>> be added to the LR. However Linux might still refuse to call the
>> interrupt handler for some reason. You might want to add a printk in the
>> Linux generic interrupt handler, probably
>> kernel/irq/chip.c:handle_fasteoi_irq in your case.
>
> The pl011 console is working now. I did not fix any thing specific
> except I cleaned up the debug code which I had added. I am not sure if
> some of my debug code was causing the issue.
>
> After cleaning up the debug code and building the code afresh, I
> started receiving the interrupts and input/output on the console is
> working fine. Now I am able to execute commands on the console.
>
I believe since I was using printk in the pl011 console driver code
for debugging, that might be causing some infinite loop. Once I
replaced all printks with HYPERVISOR_console_io(), it started working.
Later I compiled out the debug code, and it was working fine.

Regards,
Bhupinder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Wed, 15 Feb 2017, Bhupinder Thakur wrote:
> On 15 February 2017 at 13:45, Bhupinder Thakur
> <bhupinder.thakur@linaro.org> wrote:
> > Hi Stefano,
> >
> > On 14 February 2017 at 03:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >> On Mon, 13 Feb 2017, Bhupinder Thakur wrote:
> >>> Hi Stefano,
> >>>
> >>> On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
> >>> >> Hi Julien,
> >>> >>
> >>> >> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
> >>> >> > So if I understand correctly, you don't receive anymore output. Correct?
> >>> >> > Have you tried to see whether the pl011 driver is receiving interrupt or
> >>> >> > even Linux calling it.
> >>> >>
> >>> >> The pl011 driver is not receiving the interrupts (I put a
> >>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not
> >>> >> see the print on the xen console) . When the console is in this state,
> >>> >> if I keep typing the characters on the console then Xen keeps raising
> >>> >> the interrupts (for each character typed) but all those interrupts are
> >>> >> marked as inflight. So it seems that interrupt is not getting cleared
> >>> >> in GIC. Which GIC function can I look at to further debug what is the
> >>> >> issue?
> >>> >>
> >>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
> >>> >> I used to receive the interrupts all fine.
> >>> >
> >>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
> >>> > mark the interrupt as a real hardware interrupt by mistake
> >>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011
> >>> > interrupt has been added correctly to the LRs
> >>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr).
> >>>
> >>> I verified that the PL011 interrupt is getting marked pending by Xen
> >>> but guest does not receive it. I am not sure if the local interrupt
> >>> delivery is disabled on the guest because of which it may not be
> >>> receiving the interrupts. Is there a way to verify that interrupts are
> >>> indeed disabled at that point. Since the interrupt delivery works fine
> >>> when I manually read/write characters to /dev/ttyAMA0, I think there
> >>> is no issue from the Xen side in terms of raising an interrupt.
> >>>
> >>> One observation is that pl011_request_port () calls request_mem_region
> >>> () for the MMIO space allocated to pl011. The function returns -EBUSY
> >>> indicating that the memory region is already in use. But when I do cat
> >>> /proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
> >>> start of the pl011 MMIO space. I see the same error while booting the
> >>> dom0 kernel also.
> >>>
> >>> I have attached the guest boot logs when console=ttyAMA0. I have added
> >>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
> >>> and serial_core.c. The output from the console seems to suddenly stop
> >>> once the pl011 port is started. I have not added getty for ttyAMA0 in
> >>> the /etc/inittab file. I believe auto-serial-console should be able to
> >>> attach a getty session to ttyAMA0 automatically (i did try with adding
> >>> a getty command in the inittab file but that did not solve the issue).
> >>>
> >>> Towards the end of the log, we can see that every time a character is
> >>> typed, Xen raises an interrupt and the characters keep getting
> >>> collected in the IN RING buffer (in_prod keeps incrementing). Later
> >>> interrupts remain in inflight as the earlier interrupt is not cleared.
> >>
> >> Let me get this straight: who is printing all those messages we see on
> >> the guest console (xl console guest) up until "Freeing unused kernel
> >> memory"?  Is it the early pl011 driver? Or is it the Xen PV console
> >> driver?
> >>
> >> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to
> >> issue HYPERVISOR_console_io hypercalls. You might want to check that
> >> do_console_io in Xen is letting DomU calls through.
> >>
> >> Otherwise pl011_int might actually be called but the printk might not
> >> show up on the Xen console.
> >>
> >> The interrupt is enabled from Xen point of view, otherwise it would not
> >> be added to the LR. However Linux might still refuse to call the
> >> interrupt handler for some reason. You might want to add a printk in the
> >> Linux generic interrupt handler, probably
> >> kernel/irq/chip.c:handle_fasteoi_irq in your case.
> >
> > The pl011 console is working now. I did not fix any thing specific
> > except I cleaned up the debug code which I had added. I am not sure if
> > some of my debug code was causing the issue.
> >
> > After cleaning up the debug code and building the code afresh, I
> > started receiving the interrupts and input/output on the console is
> > working fine. Now I am able to execute commands on the console.
> >
> I believe since I was using printk in the pl011 console driver code
> for debugging, that might be causing some infinite loop. Once I
> replaced all printks with HYPERVISOR_console_io(), it started working.
> Later I compiled out the debug code, and it was working fine.

Yep, I tripped over that problem myself at least once or twice :-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur]

Hi,

Currently, xenconsoled supports only PV console. To support one
additional pl011 console, some of the data structures in struct domain
and certain functions such as buffer_append, ring_free_bytes, etc.
will have to be be scaled to support now two types of consoles.

Some changes are required in xenconsole client (client/maiun.c) and
tools/libxl/xl_cmdimpl.c to support one additional pl011 console.

I have done the above-mentioned changes and can now connect to both PV
and pl011 consoles.

Command for connecting to pl011 console:

xl console -t vpl011 <dom_name> connects to the pl011 console.

Currently, the console type definitions such as LIBXL_CONSOLE_TYPE_PV
are defined in /usr/include/_libxl_types.h, which is not part of the
xen tree. How should I add a new console type in this header file?

Thanks.

Regards,
Bhupinder

On 16 February 2017 at 02:51, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Wed, 15 Feb 2017, Bhupinder Thakur wrote:
>> On 15 February 2017 at 13:45, Bhupinder Thakur
>> <bhupinder.thakur@linaro.org> wrote:
>> > Hi Stefano,
>> >
>> > On 14 February 2017 at 03:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> >> On Mon, 13 Feb 2017, Bhupinder Thakur wrote:
>> >>> Hi Stefano,
>> >>>
>> >>> On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> >>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
>> >>> >> Hi Julien,
>> >>> >>
>> >>> >> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
>> >>> >> > So if I understand correctly, you don't receive anymore output. Correct?
>> >>> >> > Have you tried to see whether the pl011 driver is receiving interrupt or
>> >>> >> > even Linux calling it.
>> >>> >>
>> >>> >> The pl011 driver is not receiving the interrupts (I put a
>> >>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not
>> >>> >> see the print on the xen console) . When the console is in this state,
>> >>> >> if I keep typing the characters on the console then Xen keeps raising
>> >>> >> the interrupts (for each character typed) but all those interrupts are
>> >>> >> marked as inflight. So it seems that interrupt is not getting cleared
>> >>> >> in GIC. Which GIC function can I look at to further debug what is the
>> >>> >> issue?
>> >>> >>
>> >>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
>> >>> >> I used to receive the interrupts all fine.
>> >>> >
>> >>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
>> >>> > mark the interrupt as a real hardware interrupt by mistake
>> >>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011
>> >>> > interrupt has been added correctly to the LRs
>> >>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr).
>> >>>
>> >>> I verified that the PL011 interrupt is getting marked pending by Xen
>> >>> but guest does not receive it. I am not sure if the local interrupt
>> >>> delivery is disabled on the guest because of which it may not be
>> >>> receiving the interrupts. Is there a way to verify that interrupts are
>> >>> indeed disabled at that point. Since the interrupt delivery works fine
>> >>> when I manually read/write characters to /dev/ttyAMA0, I think there
>> >>> is no issue from the Xen side in terms of raising an interrupt.
>> >>>
>> >>> One observation is that pl011_request_port () calls request_mem_region
>> >>> () for the MMIO space allocated to pl011. The function returns -EBUSY
>> >>> indicating that the memory region is already in use. But when I do cat
>> >>> /proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
>> >>> start of the pl011 MMIO space. I see the same error while booting the
>> >>> dom0 kernel also.
>> >>>
>> >>> I have attached the guest boot logs when console=ttyAMA0. I have added
>> >>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
>> >>> and serial_core.c. The output from the console seems to suddenly stop
>> >>> once the pl011 port is started. I have not added getty for ttyAMA0 in
>> >>> the /etc/inittab file. I believe auto-serial-console should be able to
>> >>> attach a getty session to ttyAMA0 automatically (i did try with adding
>> >>> a getty command in the inittab file but that did not solve the issue).
>> >>>
>> >>> Towards the end of the log, we can see that every time a character is
>> >>> typed, Xen raises an interrupt and the characters keep getting
>> >>> collected in the IN RING buffer (in_prod keeps incrementing). Later
>> >>> interrupts remain in inflight as the earlier interrupt is not cleared.
>> >>
>> >> Let me get this straight: who is printing all those messages we see on
>> >> the guest console (xl console guest) up until "Freeing unused kernel
>> >> memory"?  Is it the early pl011 driver? Or is it the Xen PV console
>> >> driver?
>> >>
>> >> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to
>> >> issue HYPERVISOR_console_io hypercalls. You might want to check that
>> >> do_console_io in Xen is letting DomU calls through.
>> >>
>> >> Otherwise pl011_int might actually be called but the printk might not
>> >> show up on the Xen console.
>> >>
>> >> The interrupt is enabled from Xen point of view, otherwise it would not
>> >> be added to the LR. However Linux might still refuse to call the
>> >> interrupt handler for some reason. You might want to add a printk in the
>> >> Linux generic interrupt handler, probably
>> >> kernel/irq/chip.c:handle_fasteoi_irq in your case.
>> >
>> > The pl011 console is working now. I did not fix any thing specific
>> > except I cleaned up the debug code which I had added. I am not sure if
>> > some of my debug code was causing the issue.
>> >
>> > After cleaning up the debug code and building the code afresh, I
>> > started receiving the interrupts and input/output on the console is
>> > working fine. Now I am able to execute commands on the console.
>> >
>> I believe since I was using printk in the pl011 console driver code
>> for debugging, that might be causing some infinite loop. Once I
>> replaced all printks with HYPERVISOR_console_io(), it started working.
>> Later I compiled out the debug code, and it was working fine.
>
> Yep, I tripped over that problem myself at least once or twice :-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Wei Liu]

On Thu, Feb 23, 2017 at 06:45:50PM +0530, Bhupinder Thakur wrote:
> Hi,
> 
> Currently, xenconsoled supports only PV console. To support one
> additional pl011 console, some of the data structures in struct domain
> and certain functions such as buffer_append, ring_free_bytes, etc.
> will have to be be scaled to support now two types of consoles.
> 
> Some changes are required in xenconsole client (client/maiun.c) and
> tools/libxl/xl_cmdimpl.c to support one additional pl011 console.
> 

Note that we're in the process of splitting xl out of libxl directory.
Please keep an eye on xen.git staging branch.

> I have done the above-mentioned changes and can now connect to both PV
> and pl011 consoles.
> 
> Command for connecting to pl011 console:
> 
> xl console -t vpl011 <dom_name> connects to the pl011 console.
> 
> Currently, the console type definitions such as LIBXL_CONSOLE_TYPE_PV
> are defined in /usr/include/_libxl_types.h, which is not part of the
> xen tree. How should I add a new console type in this header file?
> 

See libxl_types.idl. _libxl_types.h is autogenerated from that file.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Feb 3, 2017 at 2:53 PM, Bhupinder Thakur
<bhupinder.thakur@linaro.org> wrote:
> Hi,
>
>>> Hi,
>>>
>>> I have done the changes for emulating pl011 in Xen. Currently, I have
>>> verified the emulation code by manually reading/writing data to
>>> /dev/ttyAMA0 which is the device file for pl011 device. The data is
>>> flowing fine between xenconsoled and the guest domain.
>>>
>>> As a next step, I wanted to use /dev/ttyAMA0 as a console.
>>>
>>> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
>>> "extra" directive in the domU configuration file. However, I do not
>>> see the output on the console once I attached the console using "xl
>>> console <domain_name>". I tried using "xl console -t serial
>>> <domain_name>" also but that shows the tty1 console and not the
>>> ttyAMA0 one.
>>
>> I would try to verify that when Linux uses the pl011 as a console (as
>> opposed to just writing some characters into /dev/ttyAMA0), that it
>> doesn't error out somewhere.  So like Stefano suggests, make sure you
>> check the guest kernel log after boot (assuming you can SSH or something
>> into the box) and look at 'dmesg' to see if the pl011 driver failed.
>>
>> You could also try adding some log info in you Xen pl011 emulation code
>> to see if the guest kernel driver prods the device in the sequence you
>> expect, based on looking at the linux driver for the pl011.
>
> I tested with the following changes:
>
> 1. Compiled out CONFIG_HVC_XEN
> 2. Specified "console=ttyAMA0" in the guest configuration
>
> With some more changes/fixes, the guest console is working partially
> over ttyAMA0. I can see the guest boot prints on the console. However,
> the prints stop just towards the end when the init scripts are
> executed.
>
> So it seems that during boot, initially ttyAMA0 is used as a
> write-only port, where the console driver keeps writing the output to
> the port. Till this point, only tx has been enabled on ttyAMA0 (driver
> probe() is called at this point). No irq/rx has been enabled yet. Just
> as the init scripts are executed, the irq/rx is also enabled on
> ttyAMA0 (driver startup() is called at this time which enables rx and
> registers IRQ) and then the boot prints stop.
>
> Another observation is that unless you compile out HVC_XEN support,
> the hvc0 console also keeps receiving the data even if the
> console=ttyAMA0.
>

What's your guest distro and file system settings?

This sounds to me like the console bringup job is hard-wired to
/dev/hvc0 as opposed to using the console.  I think most ubuntu
versions prior to using systemd needs the right files in /etc/init.
Check if you have /etc/init/hvc.conf or something like that and if you
have /etc/init/ttyAMA0.conf.  If you're using systemd or a different
distro, figure out what the equivalent settings there are.

-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Wed, 28 Dec 2016, Julien Grall wrote:
> Hi Stefano,
> 
> On 21/12/16 22:12, Stefano Stabellini wrote:
> > On Wed, 21 Dec 2016, Julien Grall wrote:
> > > On 20/12/2016 20:53, Stefano Stabellini wrote:
> > > > On Tue, 20 Dec 2016, Julien Grall wrote:
> > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > > > If we use hvm_params for this, we need two new hvm_params and Xen
> > > > > > needs
> > > > > > to unmap the pfn from the guest immediately, because we don't want
> > > > > > the
> > > > > > guest to have access to it.
> > > > > 
> > > > > If you unmap the pfn, the PV backend will not be able to request the
> > > > > page
> > > > > because there will be no translation available.
> > > > > 
> > > > > So what you want to do is preventing the guest to at least write into
> > > > > region
> > > > > (not sure if it is worth to restrict read)
> > > > 
> > > > That's a good idea.
> > > > 
> > > > 
> > > > > and unmap the page via the hypercall XENMEM_decrease_reservation.
> > > > 
> > > > That would be issued by the guest itself, right? To save address space?
> > > 
> > > Correct. The main use case today is ballooning, but guest could call it on
> > > any
> > > other RAM baked page.
> > > 
> > > I was thinking about more about the protection needed. Technically the
> > > data in
> > > the ring are not trusted. So if the guest is messing up with it, it would
> > > not
> > > be a big issue. Or did I miss anything here?
> > 
> > I understand that a guest would be smart to call
> > XENMEM_decrease_reservation on the PV console page for pl011, but it
> > cannot be a security measure, because, in fact, it needs to be called by
> > the guest.  Of course, a malicious guest can simply not call
> > XENMEM_decrease_reservation for it.
> 
> Sorry I was not clear. I was not suggested the guest to call
> XENMEM_decrease_reservation on ring for security but a malicious guest issuing
> the hypercall on the ring protected and replacing by another page.
> 
> This is the exact same problem as the one I mentioned on the ITS thread. The
> page live in guest memory but contains data that will only be touched by Xen.
> 
> If you remove those page from stage-2, the translation IPA -> MFN will be lost
> unless you store somewhere else. You would have to do it per-page as the
> buffer will use contiguous IPA but potentially noncontiguous MFN.
> 
> In the case of ITS the memory is provisioned by the guest. So there are not
> much to do there except adding protection in stage-2 such as write protection
> and preventing the guest to unmap it. However for the pl011 ring, as Andrew
> pointed on IRC, what we need to do is accounting this page to the domain
> memory. No mapping is necessary in stage-2.

Thanks Julien for the explanation. I think you are right.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

Hi Stefano,

On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > (CC rest maintainers for event channel questions)
> > > 
> > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > >Hi,
> > > 
> > > Hi Bhupinder,
> > > 
> > > >The idea is for Xen to act as an intermediary as shown below:
> > > >
> > > >               ring buffers
> > > >                           rx/tx fifo
> > > >dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > ><-------------------> domU
> > > >                    event
> > > >                           interrupts
> > > >
> > > >Xen will directly manage the in/out console ring buffers (allocated by
> > > >dom0 for dom0-domU console communication) for reading/writing console
> > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > >in/out console ring buffers. There should be no change in dom0 as it
> > > >will still use the same ring buffers. Similarly there should be no
> > > >change in domU which would be running a standard pll011 driver.
> > > >
> > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > >want to intercept the console events in Xen HYP which pass between
> > > >dom0 and domU. For now, I just want to capture console data coming
> > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > >interface is working.
> > > >
> > > >Since each guest domain will have a unique event channel assigned for
> > > >console communication, Xen HYP can find out the event channel for a
> > > >given domU from the start_info page of that domU, which should have
> > > 
> > > The start_info page is x86 specific. If you want to get the console
> > > event channel for ARM, you would have to use
> > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > 
> > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > in libxc/xc_dom_arm.c).
> > > 
> > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > >console event channel for a given domU. If yes and its source domain
> > > >is dom0 and destination domain is domU then it will write the data
> > > >back to the console out ring buffer of the domU and raise a console
> > > >event to dom0.
> > > >
> > > >Once this interface is working, Xen HYP can check the source and
> > > >destination dom ids and decide which way the event came from and
> > > >accordingly process the console data. To allow a mix of PV console
> > > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > >which tells whether Xen HYP should intercept and process the data (for
> > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > 
> > > I am not very familiar with the event channel code. I will let the
> > > others comment on this bit.
> > > 
> > > Regardless that, how would you decide whether the hypervisor should
> > > intercept the notification?
> > > 
> > > I can see 2 different cases:
> > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > console (or HVC then pl011)
> > > 	2) The guest is using both the PL011 and the HVC console
> > > 
> > > Should we consider the second case valid? I would say yes, because a
> > > user could specify both on the command line. If we use the same
> > > ring, the output would be a total garbage.
> > > 
> > > So maybe we need to allocate two distinct rings and event channel?
> > 
> > This sounds like the only sensible thing to me.  I think this is really
> > about adding a new device to the Xen virtual platform, and providing the
> > user the option to choose which one he wants the tool in Dom0 to be
> > presented using stdin/out. Presumably the other console/serial can be
> > redirected to a file or socket or something?
> 
> Let me explain how the PV console protocol and drivers work, because
> they are a bit unusual. 

Thanks for this.  As my detailed knowledge of Xen is relatively limited
I can only contribute with a very high level approach to which problem
we're trying to solve.


> The first PV console is advertised via
> hvm_params. The guest calls:
> 
>   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
>   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> 
> to get the two parameters to setup the ring and evtchn. If they are 0,
> the guest considers the first console unavailable. Other PV console
> rings, from the second onward, are advertised via xenstore like any
> other Xen PV protocols. In those cases, frontend and backend access
> xenstore to setup ring and event channel.

So, hvm_get_parameter does not access xenstore, because you need the
console because xenstore is available, but subsequent consoles can just
access xenstore, or did I get this completely wrong?

> 
> The PV console backends are unusual too. xenconsoled, available on all
> Xen systems, is one process per host and can handle only one PV console
> per domain. Specifically, it is only able to deal with the first console.
> Domains that have multiple PV consoles require QEMU (not as an emulator,
> but as a PV backends provider). The toolstack writes "type" =
> "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> pl011 PV consoles, but it wouldn't be the end of the world if we did.

It seems counterintuitive to run QEMU in Dom0 to serve as a backend for
emulating a device in Xen to me, but maybe I am missing the point?

Is this not a 'standard' software engineering question of choosing
between (1) expanding the functionality of xenconsoled, or (2) running
multiple instances of xenconsoled, or (3) writing a new
xenconsole_emuld, or (4) run existing huge other binary called QEMU, or
(5) something I haven't thought of ?

> 
> Additionally, Xen cannot speak xenstore. It can neither read nor write
> to it. I don't think we should add xenstore support to the hypervisor
> for this. We need to come up with a solution that doesn't require it.

So, some interface for the toolstack to tell the Xen hypervisor that
there's an event channel and a page and where they are, so that the
hypervisor can communicate with the console backend, right?

I am hoping there is *some* existing infrastructure that can be
leveraged for this and we don't have to come up with something silly
like "allocate new random hypercalls" ?

> 
> Finally, we cannot hijack one of the guest PV consoles, regardless of
> whether it's the first console or one of the others, because the guest
> can always try to use them at any time. We need a PV console reserved
> for Xen-Dom0 communications on behalf of the guest. When a VM is created
> with "pl011=y", the toolstack needs to allocate one more page and evtchn
> for the exclusive hypervisor usage.  They are not going to be advertised
> to the guest as PV consoles; otherwise, the guest could rightfully
> access them.

I suppose we should make sure that the guest cannot even touch them even
if it's lucky enough to guess the page number and event channel etc.?

> 
> Both Xen and the PV console backend need access to the two numbers (pfn
> and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> If we use hvm_params for this, we need two new hvm_params and Xen needs
> to unmap the pfn from the guest immediately, because we don't want the
> guest to have access to it.

So hvm_params is what I was hoping for above?

> 
> However, the PV console backend can access xenstore, so in that case, it
> is fine to write the pfn and evtchn of the PV console for pl011 to
> xenstore, paying attention at using the xenstore permissions
> appropriately. There are no reasons why the guest should have access to
> them; only the console backend should be able to read them. Given that
> the console backend has dom0 privileges, it is not a problem. I also
> suggest using new xenstore nodes, different from any of the existing PV
> console nodes.  For example:
> 
> /local/domain/$DOMID/xen-console/$NUM/ring-ref
> /local/domain/$DOMID/xen-console/$NUM/port
> 
> Where $DOMID is the guest domain id, and $NUM is the console number,
> starting from 0. If we use new hvm_parms for the pl011 PV console, we
> might get away without any xenstore stuff.

I'm a bit confused: Why does it make sense to write to xenstore if the
other end, which is supposed to read the values, cannot access them?  Is
this for persistence/migration or something like that?

> 
> For simplicity, given that xenconsoled doesn't support multiple PV
> consoles, we could setup the pl011 PV console *instead* of the regular
> PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> safe if the toolstack doesn't provide a PV console. When pl011 is
> requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> initial PV console. Eventually, it would be nice if xenconsoled was able
> to support both consoles at the same time.

Do you mean for development and debugging or for an existing solution?

I would advise against doing something to remove existing functionality,
since the point of this whole exercise is to improve portability of ARM
VM images, and we would not be helping with that situation by breaking
functionality of VM images that currently rely on the Xen console to be
there.

Thanks,
-Christoffer


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Stefano Stabellini]

On Tue, 20 Dec 2016, Christoffer Dall wrote:
> Hi Stefano,
> 
> On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > (CC rest maintainers for event channel questions)
> > > > 
> > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > >Hi,
> > > > 
> > > > Hi Bhupinder,
> > > > 
> > > > >The idea is for Xen to act as an intermediary as shown below:
> > > > >
> > > > >               ring buffers
> > > > >                           rx/tx fifo
> > > > >dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > > ><-------------------> domU
> > > > >                    event
> > > > >                           interrupts
> > > > >
> > > > >Xen will directly manage the in/out console ring buffers (allocated by
> > > > >dom0 for dom0-domU console communication) for reading/writing console
> > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > > >in/out console ring buffers. There should be no change in dom0 as it
> > > > >will still use the same ring buffers. Similarly there should be no
> > > > >change in domU which would be running a standard pll011 driver.
> > > > >
> > > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > >want to intercept the console events in Xen HYP which pass between
> > > > >dom0 and domU. For now, I just want to capture console data coming
> > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > >interface is working.
> > > > >
> > > > >Since each guest domain will have a unique event channel assigned for
> > > > >console communication, Xen HYP can find out the event channel for a
> > > > >given domU from the start_info page of that domU, which should have
> > > > 
> > > > The start_info page is x86 specific. If you want to get the console
> > > > event channel for ARM, you would have to use
> > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > 
> > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > in libxc/xc_dom_arm.c).
> > > > 
> > > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > > >console event channel for a given domU. If yes and its source domain
> > > > >is dom0 and destination domain is domU then it will write the data
> > > > >back to the console out ring buffer of the domU and raise a console
> > > > >event to dom0.
> > > > >
> > > > >Once this interface is working, Xen HYP can check the source and
> > > > >destination dom ids and decide which way the event came from and
> > > > >accordingly process the console data. To allow a mix of PV console
> > > > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > > >which tells whether Xen HYP should intercept and process the data (for
> > > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > > 
> > > > I am not very familiar with the event channel code. I will let the
> > > > others comment on this bit.
> > > > 
> > > > Regardless that, how would you decide whether the hypervisor should
> > > > intercept the notification?
> > > > 
> > > > I can see 2 different cases:
> > > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > > console (or HVC then pl011)
> > > > 	2) The guest is using both the PL011 and the HVC console
> > > > 
> > > > Should we consider the second case valid? I would say yes, because a
> > > > user could specify both on the command line. If we use the same
> > > > ring, the output would be a total garbage.
> > > > 
> > > > So maybe we need to allocate two distinct rings and event channel?
> > > 
> > > This sounds like the only sensible thing to me.  I think this is really
> > > about adding a new device to the Xen virtual platform, and providing the
> > > user the option to choose which one he wants the tool in Dom0 to be
> > > presented using stdin/out. Presumably the other console/serial can be
> > > redirected to a file or socket or something?
> > 
> > Let me explain how the PV console protocol and drivers work, because
> > they are a bit unusual. 
> 
> Thanks for this.  As my detailed knowledge of Xen is relatively limited
> I can only contribute with a very high level approach to which problem
> we're trying to solve.
> 
> 
> > The first PV console is advertised via
> > hvm_params. The guest calls:
> > 
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> > 
> > to get the two parameters to setup the ring and evtchn. If they are 0,
> > the guest considers the first console unavailable. Other PV console
> > rings, from the second onward, are advertised via xenstore like any
> > other Xen PV protocols. In those cases, frontend and backend access
> > xenstore to setup ring and event channel.
> 
> So, hvm_get_parameter does not access xenstore, because you need the
> console because xenstore is available, but subsequent consoles can just
> access xenstore, or did I get this completely wrong?

Right. hvm_get_parameter can be used to retrieve all sort of parameters
and doesn't access xenstore to do it. It is implemented as an hypercall
to Xen (HVMOP_get_param, AKA hvm_param). Subsequent consoles use
xenstore for the handshake in a very similar way to other PV drivers.
The frontend allocates resources and writes to xenstore. The Linux code
is: drivers/tty/hvc/hvc_xen.c:xencons_connect_backend

The data transfer protocol is always the same, but the discovery and
initial handshake are completely different.


> > The PV console backends are unusual too. xenconsoled, available on all
> > Xen systems, is one process per host and can handle only one PV console
> > per domain. Specifically, it is only able to deal with the first console.
> > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > but as a PV backends provider). The toolstack writes "type" =
> > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> 
> It seems counterintuitive to run QEMU in Dom0 to serve as a backend for
> emulating a device in Xen to me, but maybe I am missing the point?

It is indeed very counterintuitive :-D

From the days when Red Hat was still involved with Xen, we inherited a
bunch of userspace PV backends in QEMU, which have not much to do with
the rest of QEMU. Unfortunately, the QEMU build process is not flexible
enough to allow us to build a QEMU binary that supports only the PV
backends and nothing else. However, even if the binary is fat, it still
couldn't do any emulation because it is not hooked up to do it.


> Is this not a 'standard' software engineering question of choosing
> between (1) expanding the functionality of xenconsoled, or (2) running
> multiple instances of xenconsoled, or (3) writing a new
> xenconsole_emuld, or (4) run existing huge other binary called QEMU, or
> (5) something I haven't thought of ?

Yes, it is a standard sw engineering question. I think you wrote the
possible options in my order of preference.


> > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > to it. I don't think we should add xenstore support to the hypervisor
> > for this. We need to come up with a solution that doesn't require it.
> 
> So, some interface for the toolstack to tell the Xen hypervisor that
> there's an event channel and a page and where they are, so that the
> hypervisor can communicate with the console backend, right?

Yep, that's right. Either the toolstack (at domain creation) or Xen
could do the allocation.


> I am hoping there is *some* existing infrastructure that can be
> leveraged for this and we don't have to come up with something silly
> like "allocate new random hypercalls" ?

Yeah... :-)

hvm_param is very easy to use, but the guest has access to it too. If we
used hvm_param, we would need to make sure that the guest is not able to
cause any damage.

On the other hand, if we introduced a new hypercall, then we wouldn't
have to worry about the guest. But it would be another new hypercall.

Another option we would be to introduce a set of hvm_params which are
not guest-readable. Today all hvm_params are XSM_TARGET, so both "self"
and Dom0 (and stubdoms) can issue hvm_params. We could restrict a few of
them to XSM_DM_PRIV, which only allow Dom0 (and stubdoms) to issue them.
It would be as simple as changing the xsm check for a subset of them.
Obviously we would clearly document which are which.

Thoughts?


> > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > whether it's the first console or one of the others, because the guest
> > can always try to use them at any time. We need a PV console reserved
> > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > for the exclusive hypervisor usage.  They are not going to be advertised
> > to the guest as PV consoles; otherwise, the guest could rightfully
> > access them.
> 
> I suppose we should make sure that the guest cannot even touch them even
> if it's lucky enough to guess the page number and event channel etc.?

Right. But unfortunately, it is not a matter of luck, because hvm_param
is accessible to the guest.


> > Both Xen and the PV console backend need access to the two numbers (pfn
> > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > to unmap the pfn from the guest immediately, because we don't want the
> > guest to have access to it.
> 
> So hvm_params is what I was hoping for above?

See above.


> > However, the PV console backend can access xenstore, so in that case, it
> > is fine to write the pfn and evtchn of the PV console for pl011 to
> > xenstore, paying attention at using the xenstore permissions
> > appropriately. There are no reasons why the guest should have access to
> > them; only the console backend should be able to read them. Given that
> > the console backend has dom0 privileges, it is not a problem. I also
> > suggest using new xenstore nodes, different from any of the existing PV
> > console nodes.  For example:
> > 
> > /local/domain/$DOMID/xen-console/$NUM/ring-ref
> > /local/domain/$DOMID/xen-console/$NUM/port
> > 
> > Where $DOMID is the guest domain id, and $NUM is the console number,
> > starting from 0. If we use new hvm_parms for the pl011 PV console, we
> > might get away without any xenstore stuff.
> 
> I'm a bit confused: Why does it make sense to write to xenstore if the
> other end, which is supposed to read the values, cannot access them?  Is
> this for persistence/migration or something like that?

The two ends of the PV console protocol in this scenario are:
1) Xen
2) the PV console backend
Xen doesn't read xenstore, but the PV console backend does. This would
be for the benefit of the backend: it already reads parameters from
xenstore, it might be easy to pass parameters to it that way. If it is
the toolstack to do the allocation, it would be easy for it to also
write data to xenstore. However, if it is Xen to do the allocation (as
Julien suggested), this is out of the question.

In any case, it is best to use a single discovery mechanism rather than
two. This is probably unnecessary because xenconsoled could use
hvm_params or the new hypercall to get the info.


> > For simplicity, given that xenconsoled doesn't support multiple PV
> > consoles, we could setup the pl011 PV console *instead* of the regular
> > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> > safe if the toolstack doesn't provide a PV console. When pl011 is
> > requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> > initial PV console. Eventually, it would be nice if xenconsoled was able
> > to support both consoles at the same time.
> 
> Do you mean for development and debugging or for an existing solution?
> 
> I would advise against doing something to remove existing functionality,
> since the point of this whole exercise is to improve portability of ARM
> VM images, and we would not be helping with that situation by breaking
> functionality of VM images that currently rely on the Xen console to be
> there.

I mean mostly as a first milestone. However, looking at xenconsoled, it
doesn't seem difficult to add support to multiple PV consoles per
domain.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Tue, Dec 20, 2016 at 01:33:39PM -0800, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Christoffer Dall wrote:
> > Hi Stefano,
> > 
> > On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > > (CC rest maintainers for event channel questions)
> > > > > 
> > > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > > >Hi,
> > > > > 
> > > > > Hi Bhupinder,
> > > > > 
> > > > > >The idea is for Xen to act as an intermediary as shown below:
> > > > > >
> > > > > >               ring buffers
> > > > > >                           rx/tx fifo
> > > > > >dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > > > ><-------------------> domU
> > > > > >                    event
> > > > > >                           interrupts
> > > > > >
> > > > > >Xen will directly manage the in/out console ring buffers (allocated by
> > > > > >dom0 for dom0-domU console communication) for reading/writing console
> > > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > > > >in/out console ring buffers. There should be no change in dom0 as it
> > > > > >will still use the same ring buffers. Similarly there should be no
> > > > > >change in domU which would be running a standard pll011 driver.
> > > > > >
> > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > > >want to intercept the console events in Xen HYP which pass between
> > > > > >dom0 and domU. For now, I just want to capture console data coming
> > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > > >interface is working.
> > > > > >
> > > > > >Since each guest domain will have a unique event channel assigned for
> > > > > >console communication, Xen HYP can find out the event channel for a
> > > > > >given domU from the start_info page of that domU, which should have
> > > > > 
> > > > > The start_info page is x86 specific. If you want to get the console
> > > > > event channel for ARM, you would have to use
> > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > > 
> > > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > > in libxc/xc_dom_arm.c).
> > > > > 
> > > > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > > > >console event channel for a given domU. If yes and its source domain
> > > > > >is dom0 and destination domain is domU then it will write the data
> > > > > >back to the console out ring buffer of the domU and raise a console
> > > > > >event to dom0.
> > > > > >
> > > > > >Once this interface is working, Xen HYP can check the source and
> > > > > >destination dom ids and decide which way the event came from and
> > > > > >accordingly process the console data. To allow a mix of PV console
> > > > > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > > > >which tells whether Xen HYP should intercept and process the data (for
> > > > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > > > 
> > > > > I am not very familiar with the event channel code. I will let the
> > > > > others comment on this bit.
> > > > > 
> > > > > Regardless that, how would you decide whether the hypervisor should
> > > > > intercept the notification?
> > > > > 
> > > > > I can see 2 different cases:
> > > > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > > > console (or HVC then pl011)
> > > > > 	2) The guest is using both the PL011 and the HVC console
> > > > > 
> > > > > Should we consider the second case valid? I would say yes, because a
> > > > > user could specify both on the command line. If we use the same
> > > > > ring, the output would be a total garbage.
> > > > > 
> > > > > So maybe we need to allocate two distinct rings and event channel?
> > > > 
> > > > This sounds like the only sensible thing to me.  I think this is really
> > > > about adding a new device to the Xen virtual platform, and providing the
> > > > user the option to choose which one he wants the tool in Dom0 to be
> > > > presented using stdin/out. Presumably the other console/serial can be
> > > > redirected to a file or socket or something?
> > > 
> > > Let me explain how the PV console protocol and drivers work, because
> > > they are a bit unusual. 
> > 
> > Thanks for this.  As my detailed knowledge of Xen is relatively limited
> > I can only contribute with a very high level approach to which problem
> > we're trying to solve.
> > 
> > 
> > > The first PV console is advertised via
> > > hvm_params. The guest calls:
> > > 
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> > > 
> > > to get the two parameters to setup the ring and evtchn. If they are 0,
> > > the guest considers the first console unavailable. Other PV console
> > > rings, from the second onward, are advertised via xenstore like any
> > > other Xen PV protocols. In those cases, frontend and backend access
> > > xenstore to setup ring and event channel.
> > 
> > So, hvm_get_parameter does not access xenstore, because you need the
> > console because xenstore is available, but subsequent consoles can just
> > access xenstore, or did I get this completely wrong?
> 
> Right. hvm_get_parameter can be used to retrieve all sort of parameters
> and doesn't access xenstore to do it. It is implemented as an hypercall
> to Xen (HVMOP_get_param, AKA hvm_param). Subsequent consoles use
> xenstore for the handshake in a very similar way to other PV drivers.
> The frontend allocates resources and writes to xenstore. The Linux code
> is: drivers/tty/hvc/hvc_xen.c:xencons_connect_backend
> 
> The data transfer protocol is always the same, but the discovery and
> initial handshake are completely different.
> 
> 
> > > The PV console backends are unusual too. xenconsoled, available on all
> > > Xen systems, is one process per host and can handle only one PV console
> > > per domain. Specifically, it is only able to deal with the first console.
> > > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > > but as a PV backends provider). The toolstack writes "type" =
> > > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> > 
> > It seems counterintuitive to run QEMU in Dom0 to serve as a backend for
> > emulating a device in Xen to me, but maybe I am missing the point?
> 
> It is indeed very counterintuitive :-D
> 
> From the days when Red Hat was still involved with Xen, we inherited a
> bunch of userspace PV backends in QEMU, which have not much to do with
> the rest of QEMU. Unfortunately, the QEMU build process is not flexible
> enough to allow us to build a QEMU binary that supports only the PV
> backends and nothing else. However, even if the binary is fat, it still
> couldn't do any emulation because it is not hooked up to do it.
> 
> 
> > Is this not a 'standard' software engineering question of choosing
> > between (1) expanding the functionality of xenconsoled, or (2) running
> > multiple instances of xenconsoled, or (3) writing a new
> > xenconsole_emuld, or (4) run existing huge other binary called QEMU, or
> > (5) something I haven't thought of ?
> 
> Yes, it is a standard sw engineering question. I think you wrote the
> possible options in my order of preference.
> 
> 
> > > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > > to it. I don't think we should add xenstore support to the hypervisor
> > > for this. We need to come up with a solution that doesn't require it.
> > 
> > So, some interface for the toolstack to tell the Xen hypervisor that
> > there's an event channel and a page and where they are, so that the
> > hypervisor can communicate with the console backend, right?
> 
> Yep, that's right. Either the toolstack (at domain creation) or Xen
> could do the allocation.
> 
> 
> > I am hoping there is *some* existing infrastructure that can be
> > leveraged for this and we don't have to come up with something silly
> > like "allocate new random hypercalls" ?
> 
> Yeah... :-)
> 
> hvm_param is very easy to use, but the guest has access to it too. If we
> used hvm_param, we would need to make sure that the guest is not able to
> cause any damage.
> 
> On the other hand, if we introduced a new hypercall, then we wouldn't
> have to worry about the guest. But it would be another new hypercall.
> 
> Another option we would be to introduce a set of hvm_params which are
> not guest-readable. Today all hvm_params are XSM_TARGET, so both "self"
> and Dom0 (and stubdoms) can issue hvm_params. We could restrict a few of
> them to XSM_DM_PRIV, which only allow Dom0 (and stubdoms) to issue them.
> It would be as simple as changing the xsm check for a subset of them.
> Obviously we would clearly document which are which.
> 
> Thoughts?
> 

That sounds very straightforward to me (again, I haven't seen the code),
but it's no different than, say, calling write on a file descriptor you
cannot write to in Linux, so it's a known and tried concept.

> 
> > > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > > whether it's the first console or one of the others, because the guest
> > > can always try to use them at any time. We need a PV console reserved
> > > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > > for the exclusive hypervisor usage.  They are not going to be advertised
> > > to the guest as PV consoles; otherwise, the guest could rightfully
> > > access them.
> > 
> > I suppose we should make sure that the guest cannot even touch them even
> > if it's lucky enough to guess the page number and event channel etc.?
> 
> Right. But unfortunately, it is not a matter of luck, because hvm_param
> is accessible to the guest.
> 
> 
> > > Both Xen and the PV console backend need access to the two numbers (pfn
> > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > > to unmap the pfn from the guest immediately, because we don't want the
> > > guest to have access to it.
> > 
> > So hvm_params is what I was hoping for above?
> 
> See above.
> 
> 
> > > However, the PV console backend can access xenstore, so in that case, it
> > > is fine to write the pfn and evtchn of the PV console for pl011 to
> > > xenstore, paying attention at using the xenstore permissions
> > > appropriately. There are no reasons why the guest should have access to
> > > them; only the console backend should be able to read them. Given that
> > > the console backend has dom0 privileges, it is not a problem. I also
> > > suggest using new xenstore nodes, different from any of the existing PV
> > > console nodes.  For example:
> > > 
> > > /local/domain/$DOMID/xen-console/$NUM/ring-ref
> > > /local/domain/$DOMID/xen-console/$NUM/port
> > > 
> > > Where $DOMID is the guest domain id, and $NUM is the console number,
> > > starting from 0. If we use new hvm_parms for the pl011 PV console, we
> > > might get away without any xenstore stuff.
> > 
> > I'm a bit confused: Why does it make sense to write to xenstore if the
> > other end, which is supposed to read the values, cannot access them?  Is
> > this for persistence/migration or something like that?
> 
> The two ends of the PV console protocol in this scenario are:
> 1) Xen
> 2) the PV console backend
> Xen doesn't read xenstore, but the PV console backend does. This would
> be for the benefit of the backend: it already reads parameters from
> xenstore, it might be easy to pass parameters to it that way. If it is
> the toolstack to do the allocation, it would be easy for it to also
> write data to xenstore. However, if it is Xen to do the allocation (as
> Julien suggested), this is out of the question.
> 
> In any case, it is best to use a single discovery mechanism rather than
> two. This is probably unnecessary because xenconsoled could use
> hvm_params or the new hypercall to get the info.
> 
> 
> > > For simplicity, given that xenconsoled doesn't support multiple PV
> > > consoles, we could setup the pl011 PV console *instead* of the regular
> > > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> > > safe if the toolstack doesn't provide a PV console. When pl011 is
> > > requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> > > initial PV console. Eventually, it would be nice if xenconsoled was able
> > > to support both consoles at the same time.
> > 
> > Do you mean for development and debugging or for an existing solution?
> > 
> > I would advise against doing something to remove existing functionality,
> > since the point of this whole exercise is to improve portability of ARM
> > VM images, and we would not be helping with that situation by breaking
> > functionality of VM images that currently rely on the Xen console to be
> > there.
> 
> I mean mostly as a first milestone. However, looking at xenconsoled, it
> doesn't seem difficult to add support to multiple PV consoles per
> domain.

Cool!

Thanks,
-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM - Exposing a PL011 to the guest

[Julien Grall]

Hi Stefano,

On 20/12/2016 22:33, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Christoffer Dall wrote:
>> On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
>>> On Mon, 19 Dec 2016, Christoffer Dall wrote:
>>>> On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> hvm_param is very easy to use, but the guest has access to it too. If we
> used hvm_param, we would need to make sure that the guest is not able to
> cause any damage.
>
> On the other hand, if we introduced a new hypercall, then we wouldn't
> have to worry about the guest. But it would be another new hypercall.
>
> Another option we would be to introduce a set of hvm_params which are
> not guest-readable. Today all hvm_params are XSM_TARGET, so both "self"
> and Dom0 (and stubdoms) can issue hvm_params. We could restrict a few of
> them to XSM_DM_PRIV, which only allow Dom0 (and stubdoms) to issue them.
> It would be as simple as changing the xsm check for a subset of them.
> Obviously we would clearly document which are which.
>
> Thoughts?

That would work. FWIW, x86 is already restricting the access to some HVM 
parameter (see hvm_allow_set_param).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel