Adding new custom devices to Xen via QEMU

Adding new custom devices to Xen via QEMU

[Jason Dickens]

Hello,

My name is Jason Dickens and I'm a Research Scientist here at 
GrammaTech. Some of our research involves securing hypervisors and we 
have needed to add to and/or modify Xen. I have been successful in 
modifying the source for various purposes, but my question now is about 
devices. We have a custom device model implemented in QEMU which works 
great with QEMU (on Intel) standalone and with KVM, however, we now want 
access to it in Xen using the same modified QEMU build.
The only problem I seem to be having is getting Xen to send the MMIO 
R/W's to QEMU. The device is being realized, but guest access to the 
physical address range I expect to reference the device seem to go no 
place. I see in the source calls such as "register_io_handler" that 
other devices use to effect the EPT mapping. Is this what I need?
My main question is whether or not it is truly necessary to change Xen 
itself in order to introduce new devices in Xen using QEMU, or is there 
just a configuration setting? And what is the simplest way to have a 
range of physical addresses access a custom QEMU device?

Thanks,
Jason


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Wei Liu]

Hello

On Thu, Sep 29, 2016 at 09:35:21AM -0400, Jason Dickens wrote:
> Hello,
> 
> My name is Jason Dickens and I'm a Research Scientist here at GrammaTech.
> Some of our research involves securing hypervisors and we have needed to add
> to and/or modify Xen. I have been successful in modifying the source for
> various purposes, but my question now is about devices. We have a custom
> device model implemented in QEMU which works great with QEMU (on Intel)
> standalone and with KVM, however, we now want access to it in Xen using the
> same modified QEMU build.
> The only problem I seem to be having is getting Xen to send the MMIO R/W's
> to QEMU. The device is being realized, but guest access to the physical
> address range I expect to reference the device seem to go no place. I see in
> the source calls such as "register_io_handler" that other devices use to
> effect the EPT mapping. Is this what I need?
> My main question is whether or not it is truly necessary to change Xen
> itself in order to introduce new devices in Xen using QEMU, or is there just
> a configuration setting? And what is the simplest way to have a range of

It is most likely achievable by just configuring your PCI device (?) in
QEMU (provided it has a reasonable BAR size or whatnot). Check out
various device model options in xl.cfg manpage, especially
device_model_args= option allows you to add arbitrary options to QEMU.

Wei.

> physical addresses access a custom QEMU device?
> 
> Thanks,
> Jason
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Jason Dickens]

Hi Wei,

Thanks for the response. It make sense to me that if the device were on 
the PCI bus (or other such bus, e.g. USB) that it could be discovered, 
at least by an OS. Its something to consider. I should mention that our 
guest VM doesn't actually use an OS.

However, the device is not implemented that as PCI it is simply memory 
mapped. Technically, in QEMU is has type ISA because it was derived as a 
modification of the TPM device. Is it possible something is lacking in 
the QEMU model that Xen needs but KVM doesn't?
If the answer is that Xen should not need modification for any new 
devices then this gives me hope.  You've also inspired some things to 
try, like whether or not smaller modifications to the TPM device work. 
One change that is significant to mention is that the physical address 
range use is anomalous, by which I mean it not in the normal device range.

Any references you could give how Xen actually discovers when to use 
QEMU to service MMIO reads/writes would be useful too. My current 
understanding is that "something" would have to be used to configure the 
EPT to trigger QEMU to use on VMEXITs caused by such accesses.

Thanks,

Jason

P.S. Regarding  device_model_args=
I'm already using this and QEMU seems quite satisfied as the device is 
realized as expected it simply doesn't receive verified MMIO accesses 
for the device.


On 9/30/2016 7:33 AM, Wei Liu wrote:
> Hello
>
> On Thu, Sep 29, 2016 at 09:35:21AM -0400, Jason Dickens wrote:
>> Hello,
>>
>> My name is Jason Dickens and I'm a Research Scientist here at GrammaTech.
>> Some of our research involves securing hypervisors and we have needed to add
>> to and/or modify Xen. I have been successful in modifying the source for
>> various purposes, but my question now is about devices. We have a custom
>> device model implemented in QEMU which works great with QEMU (on Intel)
>> standalone and with KVM, however, we now want access to it in Xen using the
>> same modified QEMU build.
>> The only problem I seem to be having is getting Xen to send the MMIO R/W's
>> to QEMU. The device is being realized, but guest access to the physical
>> address range I expect to reference the device seem to go no place. I see in
>> the source calls such as "register_io_handler" that other devices use to
>> effect the EPT mapping. Is this what I need?
>> My main question is whether or not it is truly necessary to change Xen
>> itself in order to introduce new devices in Xen using QEMU, or is there just
>> a configuration setting? And what is the simplest way to have a range of
> It is most likely achievable by just configuring your PCI device (?) in
> QEMU (provided it has a reasonable BAR size or whatnot). Check out
> various device model options in xl.cfg manpage, especially
> device_model_args= option allows you to add arbitrary options to QEMU.
>
> Wei.
>
>> physical addresses access a custom QEMU device?
>>
>> Thanks,
>> Jason
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[David Vrabel]

On 30/09/16 14:35, Jason Dickens wrote:
> Hi Wei,
> 
> Thanks for the response. It make sense to me that if the device were on
> the PCI bus (or other such bus, e.g. USB) that it could be discovered,
> at least by an OS. Its something to consider. I should mention that our
> guest VM doesn't actually use an OS.
> 
> However, the device is not implemented that as PCI it is simply memory
> mapped. Technically, in QEMU is has type ISA because it was derived as a
> modification of the TPM device. Is it possible something is lacking in
> the QEMU model that Xen needs but KVM doesn't?
> If the answer is that Xen should not need modification for any new
> devices then this gives me hope.  You've also inspired some things to
> try, like whether or not smaller modifications to the TPM device work.
> One change that is significant to mention is that the physical address
> range use is anomalous, by which I mean it not in the normal device range.

Does device MMIO overlap with guest RAM?  If so, you'll need to
unpopulate the RAM first.

David

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Jason Dickens]

Thanks David,

This could very well be the issue, but could you please elaborate?
The questions that come up are the following:
What is the physical address range given to RAM? What range of addresses 
would work for my device?
And, if this is the case, how would I unpopulate the RAM?

There are reasons for the address chosen, and it works on other 
hypervisors (e.g. KVM) so although it might be easiest to change the 
address I really don't want to unless its the only way to keep from a 
Xen modification entirely.

Jason

On 9/30/2016 9:53 AM, David Vrabel wrote:
> On 30/09/16 14:35, Jason Dickens wrote:
>> Hi Wei,
>>
>> Thanks for the response. It make sense to me that if the device were on
>> the PCI bus (or other such bus, e.g. USB) that it could be discovered,
>> at least by an OS. Its something to consider. I should mention that our
>> guest VM doesn't actually use an OS.
>>
>> However, the device is not implemented that as PCI it is simply memory
>> mapped. Technically, in QEMU is has type ISA because it was derived as a
>> modification of the TPM device. Is it possible something is lacking in
>> the QEMU model that Xen needs but KVM doesn't?
>> If the answer is that Xen should not need modification for any new
>> devices then this gives me hope.  You've also inspired some things to
>> try, like whether or not smaller modifications to the TPM device work.
>> One change that is significant to mention is that the physical address
>> range use is anomalous, by which I mean it not in the normal device range.
> Does device MMIO overlap with guest RAM?  If so, you'll need to
> unpopulate the RAM first.
>
> David



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Konrad Rzeszutek Wilk]

On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote:
> Thanks David,
> 
> This could very well be the issue, but could you please elaborate?
> The questions that come up are the following:
> What is the physical address range given to RAM? What range of addresses
> would work for my device?

I am assuming that you implemented the emulation the same way
as other devices - that is you picked an MMIO region for your
device?

> And, if this is the case, how would I unpopulate the RAM?

See xen_ram_init. But I would just choose an region that is
most definitly in MMIO (or IO) region for your emulation.

> 
> There are reasons for the address chosen, and it works on other hypervisors
> (e.g. KVM) so although it might be easiest to change the address I really

What qemu call do you use to carve out the ranges for your device?


> don't want to unless its the only way to keep from a Xen modification
> entirely.

> 
> Jason
> 
> On 9/30/2016 9:53 AM, David Vrabel wrote:
> > On 30/09/16 14:35, Jason Dickens wrote:
> > > Hi Wei,
> > > 
> > > Thanks for the response. It make sense to me that if the device were on
> > > the PCI bus (or other such bus, e.g. USB) that it could be discovered,
> > > at least by an OS. Its something to consider. I should mention that our
> > > guest VM doesn't actually use an OS.
> > > 
> > > However, the device is not implemented that as PCI it is simply memory
> > > mapped. Technically, in QEMU is has type ISA because it was derived as a
> > > modification of the TPM device. Is it possible something is lacking in
> > > the QEMU model that Xen needs but KVM doesn't?
> > > If the answer is that Xen should not need modification for any new
> > > devices then this gives me hope.  You've also inspired some things to
> > > try, like whether or not smaller modifications to the TPM device work.
> > > One change that is significant to mention is that the physical address
> > > range use is anomalous, by which I mean it not in the normal device range.
> > Does device MMIO overlap with guest RAM?  If so, you'll need to
> > unpopulate the RAM first.
> > 
> > David
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Konrad Rzeszutek Wilk]

On Fri, Sep 30, 2016 at 03:20:09PM -0400, Jason Dickens wrote:
> Thanks Konrad,

[CC-ing Xen-devel again.]
> 
> I think you and David have successfully answered my question and pointed me
> to the key code. I have already verified that the device operates if I move
> it into the space of the TPM, but see below for reasons why I don't really
> want that.  The conclusion I'm drawing from your help is that to add a
> device where I need it,  I have to modify xen at least for areas set up in
> xen_ram_init. I've also made a few comments inline below. Its perhaps worth
> the Xen team looking at why such modification is not necessary for KVM and
> considering supporting something more automatic. I don't know but I suspect
> that for KVM, RAM is anything not overridden by a hardware device.

I don't know KVM enough to tell you.

Keep in mind that under Xen you can launch guests without QEMU. That
means the orchestration and layout of memory is not in the hands
of QEMU (like it is with KVM). Hence xen_ram_init follows the suit of
what the ABI expects (where the MMIO region is, etc).

This is all good if you have an emulated IO device (which are under 1MB)
or an emulated PCI device as they all follow the norm an allocate
themsevles in well understood locations where there are no RAM.
> 
> Jason
> 
> On 9/30/2016 2:42 PM, Konrad Rzeszutek Wilk wrote:
> > On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote:
> > > Thanks David,
> > > 
> > > This could very well be the issue, but could you please elaborate?
> > > The questions that come up are the following:
> > > What is the physical address range given to RAM? What range of addresses
> > > would work for my device?
> > I am assuming that you implemented the emulation the same way
> > as other devices - that is you picked an MMIO region for your
> > device?
> Yes its essentially the same way of choosing memory as the tpm-tis.c
> implementation.

Which AFAICT works with Xen.
> > 
> > > And, if this is the case, how would I unpopulate the RAM?
> > See xen_ram_init. But I would just choose an region that is
> > most definitly in MMIO (or IO) region for your emulation.
> As I said in a previous post there are important reasons why I need this
> device in a non-standard location. The nature the project has me searching
> for a sanitized but satisfying explanation for this post.  Its not that I
> couldn't move it, as I said above I tried the setting the address range in
> the TPM space and it worked.  I think what I'll say is the following:
> 1. Its for a proprietary, transparent, and invisible security feature.
> 2. It has to collaborate with other transparent features which help define
> its location (perhaps restrict the location is more correct).
> 
> > > There are reasons for the address chosen, and it works on other hypervisors
> > > (e.g. KVM) so although it might be easiest to change the address I really
> > What qemu call do you use to carve out the ranges for your device?
> The realization function uses:
> memory_region_add_subregion(isa_address_space(ISA_DEVICE(dev)),PORT_ADDR_BASE,
> &s->mmio);

Which should have worked? It didn't?

> > 
> > > don't want to unless its the only way to keep from a Xen modification
> > > entirely.
> > > Jason
> > > 
> > > On 9/30/2016 9:53 AM, David Vrabel wrote:
> > > > On 30/09/16 14:35, Jason Dickens wrote:
> > > > > Hi Wei,
> > > > > 
> > > > > Thanks for the response. It make sense to me that if the device were on
> > > > > the PCI bus (or other such bus, e.g. USB) that it could be discovered,
> > > > > at least by an OS. Its something to consider. I should mention that our
> > > > > guest VM doesn't actually use an OS.
> > > > > 
> > > > > However, the device is not implemented that as PCI it is simply memory
> > > > > mapped. Technically, in QEMU is has type ISA because it was derived as a
> > > > > modification of the TPM device. Is it possible something is lacking in
> > > > > the QEMU model that Xen needs but KVM doesn't?
> > > > > If the answer is that Xen should not need modification for any new
> > > > > devices then this gives me hope.  You've also inspired some things to
> > > > > try, like whether or not smaller modifications to the TPM device work.
> > > > > One change that is significant to mention is that the physical address
> > > > > range use is anomalous, by which I mean it not in the normal device range.
> > > > Does device MMIO overlap with guest RAM?  If so, you'll need to
> > > > unpopulate the RAM first.
> > > > 
> > > > David
> > > 
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > https://lists.xen.org/xen-devel
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Adding new custom devices to Xen via QEMU

[Jason Dickens]

On 9/30/2016 3:47 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 30, 2016 at 03:20:09PM -0400, Jason Dickens wrote:
>> Thanks Konrad,
> [CC-ing Xen-devel again.]
>> I think you and David have successfully answered my question and pointed me
>> to the key code. I have already verified that the device operates if I move
>> it into the space of the TPM, but see below for reasons why I don't really
>> want that.  The conclusion I'm drawing from your help is that to add a
>> device where I need it,  I have to modify xen at least for areas set up in
>> xen_ram_init. I've also made a few comments inline below. Its perhaps worth
>> the Xen team looking at why such modification is not necessary for KVM and
>> considering supporting something more automatic. I don't know but I suspect
>> that for KVM, RAM is anything not overridden by a hardware device.
> I don't know KVM enough to tell you.
>
> Keep in mind that under Xen you can launch guests without QEMU. That
> means the orchestration and layout of memory is not in the hands
> of QEMU (like it is with KVM). Hence xen_ram_init follows the suit of
> what the ABI expects (where the MMIO region is, etc).
It makes sense. I;m aware we are doing very non-standard things and it 
didn't surprise me that it would expect a device where I put it.
>
> This is all good if you have an emulated IO device (which are under 1MB)
> or an emulated PCI device as they all follow the norm an allocate
> themsevles in well understood locations where there are no RAM.
>> Jason
>>
>> On 9/30/2016 2:42 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote:
>>>> Thanks David,
>>>>
>>>> This could very well be the issue, but could you please elaborate?
>>>> The questions that come up are the following:
>>>> What is the physical address range given to RAM? What range of addresses
>>>> would work for my device?
>>> I am assuming that you implemented the emulation the same way
>>> as other devices - that is you picked an MMIO region for your
>>> device?
>> Yes its essentially the same way of choosing memory as the tpm-tis.c
>> implementation.
> Which AFAICT works with Xen.
Actually, I think it was originally developed for a custom Xen 
implementation. And yes, it works.
>>>> And, if this is the case, how would I unpopulate the RAM?
>>> See xen_ram_init. But I would just choose an region that is
>>> most definitly in MMIO (or IO) region for your emulation.
>> As I said in a previous post there are important reasons why I need this
>> device in a non-standard location. The nature the project has me searching
>> for a sanitized but satisfying explanation for this post.  Its not that I
>> couldn't move it, as I said above I tried the setting the address range in
>> the TPM space and it worked.  I think what I'll say is the following:
>> 1. Its for a proprietary, transparent, and invisible security feature.
>> 2. It has to collaborate with other transparent features which help define
>> its location (perhaps restrict the location is more correct).
>>
>>>> There are reasons for the address chosen, and it works on other hypervisors
>>>> (e.g. KVM) so although it might be easiest to change the address I really
>>> What qemu call do you use to carve out the ranges for your device?
>> The realization function uses:
>> memory_region_add_subregion(isa_address_space(ISA_DEVICE(dev)),PORT_ADDR_BASE,
>> &s->mmio);
> Which should have worked? It didn't?
Of course this does work for the normal device space. It all depends on 
what PORT_ADDR_BASE is set to whether it works or not. I think David's 
comment about overlapping with RAM was correct in our case, and the 
primary problem.
>
>>>> don't want to unless its the only way to keep from a Xen modification
>>>> entirely.
>>>> Jason
>>>>
>>>> On 9/30/2016 9:53 AM, David Vrabel wrote:
>>>>> On 30/09/16 14:35, Jason Dickens wrote:
>>>>>> Hi Wei,
>>>>>>
>>>>>> Thanks for the response. It make sense to me that if the device were on
>>>>>> the PCI bus (or other such bus, e.g. USB) that it could be discovered,
>>>>>> at least by an OS. Its something to consider. I should mention that our
>>>>>> guest VM doesn't actually use an OS.
>>>>>>
>>>>>> However, the device is not implemented that as PCI it is simply memory
>>>>>> mapped. Technically, in QEMU is has type ISA because it was derived as a
>>>>>> modification of the TPM device. Is it possible something is lacking in
>>>>>> the QEMU model that Xen needs but KVM doesn't?
>>>>>> If the answer is that Xen should not need modification for any new
>>>>>> devices then this gives me hope.  You've also inspired some things to
>>>>>> try, like whether or not smaller modifications to the TPM device work.
>>>>>> One change that is significant to mention is that the physical address
>>>>>> range use is anomalous, by which I mean it not in the normal device range.
>>>>> Does device MMIO overlap with guest RAM?  If so, you'll need to
>>>>> unpopulate the RAM first.
>>>>>
>>>>> David
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> https://lists.xen.org/xen-devel
>>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel