* Adding new custom devices to Xen via QEMU @ 2016-09-29 13:35 Jason Dickens 2016-09-30 11:33 ` Wei Liu 0 siblings, 1 reply; 8+ messages in thread From: Jason Dickens @ 2016-09-29 13:35 UTC (permalink / raw) To: xen-devel Hello, My name is Jason Dickens and I'm a Research Scientist here at GrammaTech. Some of our research involves securing hypervisors and we have needed to add to and/or modify Xen. I have been successful in modifying the source for various purposes, but my question now is about devices. We have a custom device model implemented in QEMU which works great with QEMU (on Intel) standalone and with KVM, however, we now want access to it in Xen using the same modified QEMU build. The only problem I seem to be having is getting Xen to send the MMIO R/W's to QEMU. The device is being realized, but guest access to the physical address range I expect to reference the device seem to go no place. I see in the source calls such as "register_io_handler" that other devices use to effect the EPT mapping. Is this what I need? My main question is whether or not it is truly necessary to change Xen itself in order to introduce new devices in Xen using QEMU, or is there just a configuration setting? And what is the simplest way to have a range of physical addresses access a custom QEMU device? Thanks, Jason _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-29 13:35 Adding new custom devices to Xen via QEMU Jason Dickens @ 2016-09-30 11:33 ` Wei Liu 2016-09-30 13:35 ` Jason Dickens 0 siblings, 1 reply; 8+ messages in thread From: Wei Liu @ 2016-09-30 11:33 UTC (permalink / raw) To: Jason Dickens; +Cc: Wei Liu, xen-devel Hello On Thu, Sep 29, 2016 at 09:35:21AM -0400, Jason Dickens wrote: > Hello, > > My name is Jason Dickens and I'm a Research Scientist here at GrammaTech. > Some of our research involves securing hypervisors and we have needed to add > to and/or modify Xen. I have been successful in modifying the source for > various purposes, but my question now is about devices. We have a custom > device model implemented in QEMU which works great with QEMU (on Intel) > standalone and with KVM, however, we now want access to it in Xen using the > same modified QEMU build. > The only problem I seem to be having is getting Xen to send the MMIO R/W's > to QEMU. The device is being realized, but guest access to the physical > address range I expect to reference the device seem to go no place. I see in > the source calls such as "register_io_handler" that other devices use to > effect the EPT mapping. Is this what I need? > My main question is whether or not it is truly necessary to change Xen > itself in order to introduce new devices in Xen using QEMU, or is there just > a configuration setting? And what is the simplest way to have a range of It is most likely achievable by just configuring your PCI device (?) in QEMU (provided it has a reasonable BAR size or whatnot). Check out various device model options in xl.cfg manpage, especially device_model_args= option allows you to add arbitrary options to QEMU. Wei. > physical addresses access a custom QEMU device? > > Thanks, > Jason > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-30 11:33 ` Wei Liu @ 2016-09-30 13:35 ` Jason Dickens 2016-09-30 13:53 ` David Vrabel 0 siblings, 1 reply; 8+ messages in thread From: Jason Dickens @ 2016-09-30 13:35 UTC (permalink / raw) To: Wei Liu; +Cc: xen-devel Hi Wei, Thanks for the response. It make sense to me that if the device were on the PCI bus (or other such bus, e.g. USB) that it could be discovered, at least by an OS. Its something to consider. I should mention that our guest VM doesn't actually use an OS. However, the device is not implemented that as PCI it is simply memory mapped. Technically, in QEMU is has type ISA because it was derived as a modification of the TPM device. Is it possible something is lacking in the QEMU model that Xen needs but KVM doesn't? If the answer is that Xen should not need modification for any new devices then this gives me hope. You've also inspired some things to try, like whether or not smaller modifications to the TPM device work. One change that is significant to mention is that the physical address range use is anomalous, by which I mean it not in the normal device range. Any references you could give how Xen actually discovers when to use QEMU to service MMIO reads/writes would be useful too. My current understanding is that "something" would have to be used to configure the EPT to trigger QEMU to use on VMEXITs caused by such accesses. Thanks, Jason P.S. Regarding device_model_args= I'm already using this and QEMU seems quite satisfied as the device is realized as expected it simply doesn't receive verified MMIO accesses for the device. On 9/30/2016 7:33 AM, Wei Liu wrote: > Hello > > On Thu, Sep 29, 2016 at 09:35:21AM -0400, Jason Dickens wrote: >> Hello, >> >> My name is Jason Dickens and I'm a Research Scientist here at GrammaTech. >> Some of our research involves securing hypervisors and we have needed to add >> to and/or modify Xen. I have been successful in modifying the source for >> various purposes, but my question now is about devices. We have a custom >> device model implemented in QEMU which works great with QEMU (on Intel) >> standalone and with KVM, however, we now want access to it in Xen using the >> same modified QEMU build. >> The only problem I seem to be having is getting Xen to send the MMIO R/W's >> to QEMU. The device is being realized, but guest access to the physical >> address range I expect to reference the device seem to go no place. I see in >> the source calls such as "register_io_handler" that other devices use to >> effect the EPT mapping. Is this what I need? >> My main question is whether or not it is truly necessary to change Xen >> itself in order to introduce new devices in Xen using QEMU, or is there just >> a configuration setting? And what is the simplest way to have a range of > It is most likely achievable by just configuring your PCI device (?) in > QEMU (provided it has a reasonable BAR size or whatnot). Check out > various device model options in xl.cfg manpage, especially > device_model_args= option allows you to add arbitrary options to QEMU. > > Wei. > >> physical addresses access a custom QEMU device? >> >> Thanks, >> Jason >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> https://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-30 13:35 ` Jason Dickens @ 2016-09-30 13:53 ` David Vrabel 2016-09-30 14:29 ` Jason Dickens 0 siblings, 1 reply; 8+ messages in thread From: David Vrabel @ 2016-09-30 13:53 UTC (permalink / raw) To: Jason Dickens, Wei Liu; +Cc: xen-devel On 30/09/16 14:35, Jason Dickens wrote: > Hi Wei, > > Thanks for the response. It make sense to me that if the device were on > the PCI bus (or other such bus, e.g. USB) that it could be discovered, > at least by an OS. Its something to consider. I should mention that our > guest VM doesn't actually use an OS. > > However, the device is not implemented that as PCI it is simply memory > mapped. Technically, in QEMU is has type ISA because it was derived as a > modification of the TPM device. Is it possible something is lacking in > the QEMU model that Xen needs but KVM doesn't? > If the answer is that Xen should not need modification for any new > devices then this gives me hope. You've also inspired some things to > try, like whether or not smaller modifications to the TPM device work. > One change that is significant to mention is that the physical address > range use is anomalous, by which I mean it not in the normal device range. Does device MMIO overlap with guest RAM? If so, you'll need to unpopulate the RAM first. David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-30 13:53 ` David Vrabel @ 2016-09-30 14:29 ` Jason Dickens 2016-09-30 18:42 ` Konrad Rzeszutek Wilk 0 siblings, 1 reply; 8+ messages in thread From: Jason Dickens @ 2016-09-30 14:29 UTC (permalink / raw) To: David Vrabel, Wei Liu; +Cc: xen-devel Thanks David, This could very well be the issue, but could you please elaborate? The questions that come up are the following: What is the physical address range given to RAM? What range of addresses would work for my device? And, if this is the case, how would I unpopulate the RAM? There are reasons for the address chosen, and it works on other hypervisors (e.g. KVM) so although it might be easiest to change the address I really don't want to unless its the only way to keep from a Xen modification entirely. Jason On 9/30/2016 9:53 AM, David Vrabel wrote: > On 30/09/16 14:35, Jason Dickens wrote: >> Hi Wei, >> >> Thanks for the response. It make sense to me that if the device were on >> the PCI bus (or other such bus, e.g. USB) that it could be discovered, >> at least by an OS. Its something to consider. I should mention that our >> guest VM doesn't actually use an OS. >> >> However, the device is not implemented that as PCI it is simply memory >> mapped. Technically, in QEMU is has type ISA because it was derived as a >> modification of the TPM device. Is it possible something is lacking in >> the QEMU model that Xen needs but KVM doesn't? >> If the answer is that Xen should not need modification for any new >> devices then this gives me hope. You've also inspired some things to >> try, like whether or not smaller modifications to the TPM device work. >> One change that is significant to mention is that the physical address >> range use is anomalous, by which I mean it not in the normal device range. > Does device MMIO overlap with guest RAM? If so, you'll need to > unpopulate the RAM first. > > David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-30 14:29 ` Jason Dickens @ 2016-09-30 18:42 ` Konrad Rzeszutek Wilk [not found] ` <b980347d-d77b-d33e-4de9-8ed2f042c136@grammatech.com> 0 siblings, 1 reply; 8+ messages in thread From: Konrad Rzeszutek Wilk @ 2016-09-30 18:42 UTC (permalink / raw) To: Jason Dickens; +Cc: Wei Liu, David Vrabel, xen-devel On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote: > Thanks David, > > This could very well be the issue, but could you please elaborate? > The questions that come up are the following: > What is the physical address range given to RAM? What range of addresses > would work for my device? I am assuming that you implemented the emulation the same way as other devices - that is you picked an MMIO region for your device? > And, if this is the case, how would I unpopulate the RAM? See xen_ram_init. But I would just choose an region that is most definitly in MMIO (or IO) region for your emulation. > > There are reasons for the address chosen, and it works on other hypervisors > (e.g. KVM) so although it might be easiest to change the address I really What qemu call do you use to carve out the ranges for your device? > don't want to unless its the only way to keep from a Xen modification > entirely. > > Jason > > On 9/30/2016 9:53 AM, David Vrabel wrote: > > On 30/09/16 14:35, Jason Dickens wrote: > > > Hi Wei, > > > > > > Thanks for the response. It make sense to me that if the device were on > > > the PCI bus (or other such bus, e.g. USB) that it could be discovered, > > > at least by an OS. Its something to consider. I should mention that our > > > guest VM doesn't actually use an OS. > > > > > > However, the device is not implemented that as PCI it is simply memory > > > mapped. Technically, in QEMU is has type ISA because it was derived as a > > > modification of the TPM device. Is it possible something is lacking in > > > the QEMU model that Xen needs but KVM doesn't? > > > If the answer is that Xen should not need modification for any new > > > devices then this gives me hope. You've also inspired some things to > > > try, like whether or not smaller modifications to the TPM device work. > > > One change that is significant to mention is that the physical address > > > range use is anomalous, by which I mean it not in the normal device range. > > Does device MMIO overlap with guest RAM? If so, you'll need to > > unpopulate the RAM first. > > > > David > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <b980347d-d77b-d33e-4de9-8ed2f042c136@grammatech.com>]
* Re: Adding new custom devices to Xen via QEMU [not found] ` <b980347d-d77b-d33e-4de9-8ed2f042c136@grammatech.com> @ 2016-09-30 19:47 ` Konrad Rzeszutek Wilk 2016-09-30 20:06 ` Jason Dickens 0 siblings, 1 reply; 8+ messages in thread From: Konrad Rzeszutek Wilk @ 2016-09-30 19:47 UTC (permalink / raw) To: Jason Dickens, xen-devel On Fri, Sep 30, 2016 at 03:20:09PM -0400, Jason Dickens wrote: > Thanks Konrad, [CC-ing Xen-devel again.] > > I think you and David have successfully answered my question and pointed me > to the key code. I have already verified that the device operates if I move > it into the space of the TPM, but see below for reasons why I don't really > want that. The conclusion I'm drawing from your help is that to add a > device where I need it, I have to modify xen at least for areas set up in > xen_ram_init. I've also made a few comments inline below. Its perhaps worth > the Xen team looking at why such modification is not necessary for KVM and > considering supporting something more automatic. I don't know but I suspect > that for KVM, RAM is anything not overridden by a hardware device. I don't know KVM enough to tell you. Keep in mind that under Xen you can launch guests without QEMU. That means the orchestration and layout of memory is not in the hands of QEMU (like it is with KVM). Hence xen_ram_init follows the suit of what the ABI expects (where the MMIO region is, etc). This is all good if you have an emulated IO device (which are under 1MB) or an emulated PCI device as they all follow the norm an allocate themsevles in well understood locations where there are no RAM. > > Jason > > On 9/30/2016 2:42 PM, Konrad Rzeszutek Wilk wrote: > > On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote: > > > Thanks David, > > > > > > This could very well be the issue, but could you please elaborate? > > > The questions that come up are the following: > > > What is the physical address range given to RAM? What range of addresses > > > would work for my device? > > I am assuming that you implemented the emulation the same way > > as other devices - that is you picked an MMIO region for your > > device? > Yes its essentially the same way of choosing memory as the tpm-tis.c > implementation. Which AFAICT works with Xen. > > > > > And, if this is the case, how would I unpopulate the RAM? > > See xen_ram_init. But I would just choose an region that is > > most definitly in MMIO (or IO) region for your emulation. > As I said in a previous post there are important reasons why I need this > device in a non-standard location. The nature the project has me searching > for a sanitized but satisfying explanation for this post. Its not that I > couldn't move it, as I said above I tried the setting the address range in > the TPM space and it worked. I think what I'll say is the following: > 1. Its for a proprietary, transparent, and invisible security feature. > 2. It has to collaborate with other transparent features which help define > its location (perhaps restrict the location is more correct). > > > > There are reasons for the address chosen, and it works on other hypervisors > > > (e.g. KVM) so although it might be easiest to change the address I really > > What qemu call do you use to carve out the ranges for your device? > The realization function uses: > memory_region_add_subregion(isa_address_space(ISA_DEVICE(dev)),PORT_ADDR_BASE, > &s->mmio); Which should have worked? It didn't? > > > > > don't want to unless its the only way to keep from a Xen modification > > > entirely. > > > Jason > > > > > > On 9/30/2016 9:53 AM, David Vrabel wrote: > > > > On 30/09/16 14:35, Jason Dickens wrote: > > > > > Hi Wei, > > > > > > > > > > Thanks for the response. It make sense to me that if the device were on > > > > > the PCI bus (or other such bus, e.g. USB) that it could be discovered, > > > > > at least by an OS. Its something to consider. I should mention that our > > > > > guest VM doesn't actually use an OS. > > > > > > > > > > However, the device is not implemented that as PCI it is simply memory > > > > > mapped. Technically, in QEMU is has type ISA because it was derived as a > > > > > modification of the TPM device. Is it possible something is lacking in > > > > > the QEMU model that Xen needs but KVM doesn't? > > > > > If the answer is that Xen should not need modification for any new > > > > > devices then this gives me hope. You've also inspired some things to > > > > > try, like whether or not smaller modifications to the TPM device work. > > > > > One change that is significant to mention is that the physical address > > > > > range use is anomalous, by which I mean it not in the normal device range. > > > > Does device MMIO overlap with guest RAM? If so, you'll need to > > > > unpopulate the RAM first. > > > > > > > > David > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > https://lists.xen.org/xen-devel > > _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding new custom devices to Xen via QEMU 2016-09-30 19:47 ` Konrad Rzeszutek Wilk @ 2016-09-30 20:06 ` Jason Dickens 0 siblings, 0 replies; 8+ messages in thread From: Jason Dickens @ 2016-09-30 20:06 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, xen-devel On 9/30/2016 3:47 PM, Konrad Rzeszutek Wilk wrote: > On Fri, Sep 30, 2016 at 03:20:09PM -0400, Jason Dickens wrote: >> Thanks Konrad, > [CC-ing Xen-devel again.] >> I think you and David have successfully answered my question and pointed me >> to the key code. I have already verified that the device operates if I move >> it into the space of the TPM, but see below for reasons why I don't really >> want that. The conclusion I'm drawing from your help is that to add a >> device where I need it, I have to modify xen at least for areas set up in >> xen_ram_init. I've also made a few comments inline below. Its perhaps worth >> the Xen team looking at why such modification is not necessary for KVM and >> considering supporting something more automatic. I don't know but I suspect >> that for KVM, RAM is anything not overridden by a hardware device. > I don't know KVM enough to tell you. > > Keep in mind that under Xen you can launch guests without QEMU. That > means the orchestration and layout of memory is not in the hands > of QEMU (like it is with KVM). Hence xen_ram_init follows the suit of > what the ABI expects (where the MMIO region is, etc). It makes sense. I;m aware we are doing very non-standard things and it didn't surprise me that it would expect a device where I put it. > > This is all good if you have an emulated IO device (which are under 1MB) > or an emulated PCI device as they all follow the norm an allocate > themsevles in well understood locations where there are no RAM. >> Jason >> >> On 9/30/2016 2:42 PM, Konrad Rzeszutek Wilk wrote: >>> On Fri, Sep 30, 2016 at 10:29:20AM -0400, Jason Dickens wrote: >>>> Thanks David, >>>> >>>> This could very well be the issue, but could you please elaborate? >>>> The questions that come up are the following: >>>> What is the physical address range given to RAM? What range of addresses >>>> would work for my device? >>> I am assuming that you implemented the emulation the same way >>> as other devices - that is you picked an MMIO region for your >>> device? >> Yes its essentially the same way of choosing memory as the tpm-tis.c >> implementation. > Which AFAICT works with Xen. Actually, I think it was originally developed for a custom Xen implementation. And yes, it works. >>>> And, if this is the case, how would I unpopulate the RAM? >>> See xen_ram_init. But I would just choose an region that is >>> most definitly in MMIO (or IO) region for your emulation. >> As I said in a previous post there are important reasons why I need this >> device in a non-standard location. The nature the project has me searching >> for a sanitized but satisfying explanation for this post. Its not that I >> couldn't move it, as I said above I tried the setting the address range in >> the TPM space and it worked. I think what I'll say is the following: >> 1. Its for a proprietary, transparent, and invisible security feature. >> 2. It has to collaborate with other transparent features which help define >> its location (perhaps restrict the location is more correct). >> >>>> There are reasons for the address chosen, and it works on other hypervisors >>>> (e.g. KVM) so although it might be easiest to change the address I really >>> What qemu call do you use to carve out the ranges for your device? >> The realization function uses: >> memory_region_add_subregion(isa_address_space(ISA_DEVICE(dev)),PORT_ADDR_BASE, >> &s->mmio); > Which should have worked? It didn't? Of course this does work for the normal device space. It all depends on what PORT_ADDR_BASE is set to whether it works or not. I think David's comment about overlapping with RAM was correct in our case, and the primary problem. > >>>> don't want to unless its the only way to keep from a Xen modification >>>> entirely. >>>> Jason >>>> >>>> On 9/30/2016 9:53 AM, David Vrabel wrote: >>>>> On 30/09/16 14:35, Jason Dickens wrote: >>>>>> Hi Wei, >>>>>> >>>>>> Thanks for the response. It make sense to me that if the device were on >>>>>> the PCI bus (or other such bus, e.g. USB) that it could be discovered, >>>>>> at least by an OS. Its something to consider. I should mention that our >>>>>> guest VM doesn't actually use an OS. >>>>>> >>>>>> However, the device is not implemented that as PCI it is simply memory >>>>>> mapped. Technically, in QEMU is has type ISA because it was derived as a >>>>>> modification of the TPM device. Is it possible something is lacking in >>>>>> the QEMU model that Xen needs but KVM doesn't? >>>>>> If the answer is that Xen should not need modification for any new >>>>>> devices then this gives me hope. You've also inspired some things to >>>>>> try, like whether or not smaller modifications to the TPM device work. >>>>>> One change that is significant to mention is that the physical address >>>>>> range use is anomalous, by which I mean it not in the normal device range. >>>>> Does device MMIO overlap with guest RAM? If so, you'll need to >>>>> unpopulate the RAM first. >>>>> >>>>> David >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> https://lists.xen.org/xen-devel >> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-09-30 20:06 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-09-29 13:35 Adding new custom devices to Xen via QEMU Jason Dickens 2016-09-30 11:33 ` Wei Liu 2016-09-30 13:35 ` Jason Dickens 2016-09-30 13:53 ` David Vrabel 2016-09-30 14:29 ` Jason Dickens 2016-09-30 18:42 ` Konrad Rzeszutek Wilk [not found] ` <b980347d-d77b-d33e-4de9-8ed2f042c136@grammatech.com> 2016-09-30 19:47 ` Konrad Rzeszutek Wilk 2016-09-30 20:06 ` Jason Dickens
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.