[Qemu-devel] "Using Python to investigate EFI and ACPI"

[Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

Hi Josh,

I just found the LWN.net article with the title in the subject:

  http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/

("Thank you LWN", of course. :))

This got me kind of excited. In your presentation, according to the
article, you were using KVM and OVMF to showcase the ACPI features of
"BITS". Further, in

  http://biosbits.org/scripting/

you write, "You can also find arbitrary ACPI objects by path, and
evaluate arbitrary ACPI methods, with the return value decoded into a
Python object".

I'm excited because for virtualization we might have the reverse use
case: use BITS to test / debug / develop the ACPI generator of QEMU. :)

As you may now, QEMU generates almost all of its ACPI tables
dynamically, providing the guest with an "ACPI payload" that describes
the actual virtual hardware as accurately as possible.

Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
"fw_cfg" (which is a simple protocol, comprising, at this point, one
selector and one data register, which are IO ports or MMIO locations --
see "docs/specs/fw_cfg.txt" in QEMU and
"Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).

The guest firmware downloads the "ACPI payload" over fw_cfg, does some
generic processing on them (we call this the "ACPI linker/loader
script"), and ultimately installs the ACPI tables in a way that is
appropriate for the firmware in question.

To be more concrete, the tables you investigated in your presentation
with BITS were generated (very roughly) by the following files in QEMU:

  hw/acpi/aml-build.c
  hw/acpi/bios-linker-loader.c
  hw/acpi/...
  hw/i386/acpi-build.c

and processed and installed by the following source file in edk2:

  OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c

With this background, you can probably see where I'm going with this. It
is not really easy to *test* the AML methods that QEMU generates
(piecing them together, with helper functions, from AML primitives),
without dedicated guest kernel drivers. I think the only method that I
know of is the following:

- in the Linux guest, dump the ACPI tables with acpidump, or save them
from sysfs directly (/sys/firmware/acpi/tables)
- pass the DSDT and the SSDTs (and any data tables referenced by them?)
to AcpiExec from the ACPICA suite
- work with AcpiExec

But, for simple testing, can we maybe run your tool within the guest,
before the runtime OS boots?

Although AcpiDump is being ported to edk2:

  http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109

I'm unsure if it will be able to evaluate / execute AML methods. (If it
parallels the "acpidump" utility we have on GNU/Linux distros, it won't.)

Thus it would be awesome if we had some AcpiExec-like functionality
early on in the guest (for example in the form of a UEFI Shell
application, or as a python tool that runs within the edk2 Python port,
or even in grub).

For example, assume your runtime guest OS is Windows (with its picky,
closed-source ACPI interpreter); you make a change in QEMU's ACPI
generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
verify the change "by eye", exit the shell, and *then* continue booting
Windows. (Which will hopefully not BSOD at that point, after the
verification with BITS / AcpiExec etc.)

So, I have three questions:

(1) What is the relationship between the ACPI facility of BITS, and ACPICA?

(2) Is there a bit more comprehensive documentation about the ACPI
module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
with their incredible documentation (documents/acpica-reference.pdf). It
would be great if BITS' ACPI module had a list of commands, to see what
is there to play with.

(3) I should mention that QEMU generates ACPI also for arm/aarch64
virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same edk2
module as noted above (built for arm/aarch64) does the guest side
processing. Do you think it's possible to use BITS in arm/aarch64 VMs?

... I apologize if tools / documentation already exist for this kind of
development work; everyone please educate me then. I hope my questions
make at least some sense; I realize this email isn't well organized.

Thanks
Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

Side point:

On 09/03/15 11:16, Laszlo Ersek wrote:
> 
> I just found the LWN.net article with the title in the subject:
> 
>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> 
> ("Thank you LWN", of course. :))

I've just been advised in private, well-meaningly, that I shouldn't post
LWN Subscriber Links publicly.

I did that consciously. The notice on LWN's Subscriber Link generator
page is *not* lost on me; however, Jonathan Corbet has previously
publicly approved such *occasional* posting:

https://www.reddit.com/r/linux/comments/2lfhh8/kdbus_meets_linuxkernel_lwnnet/clumuze

I thought making this one article publicly readable "early" was (a)
basis for this discussion, (b) showcasing the value of LWN's journalism
(ie. "advertisment").

Thanks
Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Jake Edge]

On Thu, 3 Sep 2015 11:41:57 +0200 Laszlo Ersek wrote:
> Side point:
> 
> On 09/03/15 11:16, Laszlo Ersek wrote:
> > 
> > I just found the LWN.net article with the title in the subject:
> > 
> >   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> > 
> > ("Thank you LWN", of course. :))
> 
> I've just been advised in private, well-meaningly, that I shouldn't post
> LWN Subscriber Links publicly.

fwiw, I see no problem with this ... it is what we have envisioned
subscriber links being used for ...

please enjoy!

jake

-- 
Jake Edge - LWN - jake@lwn.net - http://lwn.net

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Josh Triplett]

On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> Hi Josh,
> 
> I just found the LWN.net article with the title in the subject:
> 
>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> 
> ("Thank you LWN", of course. :))
> 
> This got me kind of excited. In your presentation, according to the
> article, you were using KVM and OVMF to showcase the ACPI features of
> "BITS". Further, in
> 
>   http://biosbits.org/scripting/

(Note: that page is badly out of date, by a couple of years, and we need
to drop it in favor of the various more recent presentations.  All of
that functionality still exists, but we have an interactive Python REPL
now, and the ACPI methods have moved to the "acpi" module and become
easier to use.)

> you write, "You can also find arbitrary ACPI objects by path, and
> evaluate arbitrary ACPI methods, with the return value decoded into a
> Python object".

Right.  In particular, you can call acpi.get_objpaths to list the ACPI
tree, and acpi.evaluate to evaluate an arbitrary method.

> I'm excited because for virtualization we might have the reverse use
> case: use BITS to test / debug / develop the ACPI generator of QEMU. :)

Awesome.  Unit testing and CI for QEMU sounds like an ideal use case for
BITS.

> As you may now, QEMU generates almost all of its ACPI tables
> dynamically, providing the guest with an "ACPI payload" that describes
> the actual virtual hardware as accurately as possible.

Many BIOSes do that, since the hardware configuration may vary at
runtime.

> Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
> "fw_cfg" (which is a simple protocol, comprising, at this point, one
> selector and one data register, which are IO ports or MMIO locations --
> see "docs/specs/fw_cfg.txt" in QEMU and
> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).

Interesting; I hadn't seen that protocol before.

Do you virtualize those I/O ports by CPU, to make them thread-safe, or
does the last address written to 0x510 get saved system-wide, making it
unsafe for concurrent access?

> The guest firmware downloads the "ACPI payload" over fw_cfg, does some
> generic processing on them (we call this the "ACPI linker/loader
> script"), and ultimately installs the ACPI tables in a way that is
> appropriate for the firmware in question.
> 
> To be more concrete, the tables you investigated in your presentation
> with BITS were generated (very roughly) by the following files in QEMU:
> 
>   hw/acpi/aml-build.c
>   hw/acpi/bios-linker-loader.c
>   hw/acpi/...
>   hw/i386/acpi-build.c
> 
> and processed and installed by the following source file in edk2:
> 
>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
> 
> With this background, you can probably see where I'm going with this. It
> is not really easy to *test* the AML methods that QEMU generates
> (piecing them together, with helper functions, from AML primitives),
> without dedicated guest kernel drivers. I think the only method that I
> know of is the following:
> 
> - in the Linux guest, dump the ACPI tables with acpidump, or save them
> from sysfs directly (/sys/firmware/acpi/tables)
> - pass the DSDT and the SSDTs (and any data tables referenced by them?)
> to AcpiExec from the ACPICA suite
> - work with AcpiExec
> 
> But, for simple testing, can we maybe run your tool within the guest,
> before the runtime OS boots?

Yes, absolutely.  We have a batch-mode testing mechanism based on a
config file; you'd probably want to make use of that.  With some
extensions, it could dump results either to an emulated serial port or
some other interface that you can read from outside qemu.  We also need
to work on making the results more machine-parseable for automation.

> Although AcpiDump is being ported to edk2:
> 
>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
> 
> I'm unsure if it will be able to evaluate / execute AML methods. (If it
> parallels the "acpidump" utility we have on GNU/Linux distros, it won't.)

Probably not.  And I haven't heard of any plans to port acpiexec to
EDK2.

> Thus it would be awesome if we had some AcpiExec-like functionality
> early on in the guest (for example in the form of a UEFI Shell
> application, or as a python tool that runs within the edk2 Python port,
> or even in grub).
> 
> For example, assume your runtime guest OS is Windows (with its picky,
> closed-source ACPI interpreter); you make a change in QEMU's ACPI
> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
> verify the change "by eye", exit the shell, and *then* continue booting
> Windows. (Which will hopefully not BSOD at that point, after the
> verification with BITS / AcpiExec etc.)
> 
> So, I have three questions:
> 
> (1) What is the relationship between the ACPI facility of BITS, and ACPICA?

BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA as
a git submodule and build it as part of BITS.  acpi.evaluate uses the
ACPICA interpreter.

> (2) Is there a bit more comprehensive documentation about the ACPI
> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
> with their incredible documentation (documents/acpica-reference.pdf). It
> would be great if BITS' ACPI module had a list of commands, to see what
> is there to play with.

We don't come close to the level of documentation for ACPICA, but we do
have pydoc documentation for the modules in BITS, including acpi.  You
can run help("acpi") within BITS, or read acpi.py.  We've tried to make
sure all of the methods considered "API" have docstrings.

> (3) I should mention that QEMU generates ACPI also for arm/aarch64
> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same edk2
> module as noted above (built for arm/aarch64) does the guest side
> processing. Do you think it's possible to use BITS in arm/aarch64 VMs?

Some folks from Linaro started looking into a BITS arm64 port.

> ... I apologize if tools / documentation already exist for this kind of
> development work; everyone please educate me then. I hope my questions
> make at least some sense; I realize this email isn't well organized.

Makes perfect sense, and thanks for your mail!  I love the idea of using
BITS to test qemu's own ACPI.

- Josh Triplett

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[josh]

On Thu, Sep 03, 2015 at 07:50:07AM -0700, Josh Triplett wrote:
> On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> > I'm excited because for virtualization we might have the reverse use
> > case: use BITS to test / debug / develop the ACPI generator of QEMU. :)
> 
> Awesome.  Unit testing and CI for QEMU sounds like an ideal use case for
> BITS.
[...]
> > But, for simple testing, can we maybe run your tool within the guest,
> > before the runtime OS boots?
> 
> Yes, absolutely.  We have a batch-mode testing mechanism based on a
> config file; you'd probably want to make use of that.  With some
> extensions, it could dump results either to an emulated serial port or
> some other interface that you can read from outside qemu.  We also need
> to work on making the results more machine-parseable for automation.

One other note on that front: if you write new tests for ACPI (see
python/testacpi.py), or new data structure parsers, please feel free to
contribute them back to BITS, and we'll happily run them on non-qemu
systems as well.

- Josh Triplett

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

On 09/03/15 16:50, Josh Triplett wrote:
> On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
>> Hi Josh,
>>
>> I just found the LWN.net article with the title in the subject:
>>
>>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
>>
>> ("Thank you LWN", of course. :))
>>
>> This got me kind of excited. In your presentation, according to the
>> article, you were using KVM and OVMF to showcase the ACPI features of
>> "BITS". Further, in
>>
>>   http://biosbits.org/scripting/
> 
> (Note: that page is badly out of date, by a couple of years, and we need
> to drop it in favor of the various more recent presentations.  All of
> that functionality still exists, but we have an interactive Python REPL
> now, and the ACPI methods have moved to the "acpi" module and become
> easier to use.)
> 
>> you write, "You can also find arbitrary ACPI objects by path, and
>> evaluate arbitrary ACPI methods, with the return value decoded into a
>> Python object".
> 
> Right.  In particular, you can call acpi.get_objpaths to list the ACPI
> tree, and acpi.evaluate to evaluate an arbitrary method.
> 
>> I'm excited because for virtualization we might have the reverse use
>> case: use BITS to test / debug / develop the ACPI generator of QEMU. :)
> 
> Awesome.  Unit testing and CI for QEMU sounds like an ideal use case for
> BITS.
> 
>> As you may now, QEMU generates almost all of its ACPI tables
>> dynamically, providing the guest with an "ACPI payload" that describes
>> the actual virtual hardware as accurately as possible.
> 
> Many BIOSes do that, since the hardware configuration may vary at
> runtime.
> 
>> Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
>> "fw_cfg" (which is a simple protocol, comprising, at this point, one
>> selector and one data register, which are IO ports or MMIO locations --
>> see "docs/specs/fw_cfg.txt" in QEMU and
>> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).
> 
> Interesting; I hadn't seen that protocol before.
> 
> Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> does the last address written to 0x510 get saved system-wide, making it
> unsafe for concurrent access?

I think fw_cfg is not meant to be accessed by several CPUs concurrently.
The protocol is stateful (selected key, offset within blob associated
with selected key, etc), and "accessing CPU" is not part of that state.

> 
>> The guest firmware downloads the "ACPI payload" over fw_cfg, does some
>> generic processing on them (we call this the "ACPI linker/loader
>> script"), and ultimately installs the ACPI tables in a way that is
>> appropriate for the firmware in question.
>>
>> To be more concrete, the tables you investigated in your presentation
>> with BITS were generated (very roughly) by the following files in QEMU:
>>
>>   hw/acpi/aml-build.c
>>   hw/acpi/bios-linker-loader.c
>>   hw/acpi/...
>>   hw/i386/acpi-build.c
>>
>> and processed and installed by the following source file in edk2:
>>
>>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
>>
>> With this background, you can probably see where I'm going with this. It
>> is not really easy to *test* the AML methods that QEMU generates
>> (piecing them together, with helper functions, from AML primitives),
>> without dedicated guest kernel drivers. I think the only method that I
>> know of is the following:
>>
>> - in the Linux guest, dump the ACPI tables with acpidump, or save them
>> from sysfs directly (/sys/firmware/acpi/tables)
>> - pass the DSDT and the SSDTs (and any data tables referenced by them?)
>> to AcpiExec from the ACPICA suite
>> - work with AcpiExec
>>
>> But, for simple testing, can we maybe run your tool within the guest,
>> before the runtime OS boots?
> 
> Yes, absolutely.  We have a batch-mode testing mechanism based on a
> config file; you'd probably want to make use of that.  With some
> extensions, it could dump results either to an emulated serial port or
> some other interface that you can read from outside qemu.  We also need
> to work on making the results more machine-parseable for automation.

While I certainly don't discount automation, my primary use case is
interactive development / testing. :) (Although, I can see myself
canning some commands in a script or config file, and invoking *that*
interactively. Which was your point, probably.)

> 
>> Although AcpiDump is being ported to edk2:
>>
>>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
>>
>> I'm unsure if it will be able to evaluate / execute AML methods. (If it
>> parallels the "acpidump" utility we have on GNU/Linux distros, it won't.)
> 
> Probably not.  And I haven't heard of any plans to port acpiexec to
> EDK2.
> 
>> Thus it would be awesome if we had some AcpiExec-like functionality
>> early on in the guest (for example in the form of a UEFI Shell
>> application, or as a python tool that runs within the edk2 Python port,
>> or even in grub).
>>
>> For example, assume your runtime guest OS is Windows (with its picky,
>> closed-source ACPI interpreter); you make a change in QEMU's ACPI
>> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
>> verify the change "by eye", exit the shell, and *then* continue booting
>> Windows. (Which will hopefully not BSOD at that point, after the
>> verification with BITS / AcpiExec etc.)
>>
>> So, I have three questions:
>>
>> (1) What is the relationship between the ACPI facility of BITS, and ACPICA?
> 
> BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA as
> a git submodule and build it as part of BITS.  acpi.evaluate uses the
> ACPICA interpreter.

Awesome! :)

Another question: when you execute an AML method that does, say, IO port
access, does the AML interpreter of ACPICA actually *perform* that IO
port access? Because, the one that is embedded in Linux obviously does,
and the one that is embedded in the userspace ACPICA command line
utility "acpiexec" obviously doesn't.

I assume (and very much hope) that the IO port access *is* performed
from BITS, simply because you developed it for physical machines, and it
wouldn't make much sense to avoid actual hardware access that was
implemented by the BIOS vendor for that platform.

If that is the case, then this tool could become the killer ACPI tester
for QEMU developers -- the hardware accesses in the AML methods
generated by QEMU would actually poke QEMU devices! (Unlike the
userspace "acpiexec" utility.) It would completely detach Linux guest
driver development from host side / firmware development. \o/

> 
>> (2) Is there a bit more comprehensive documentation about the ACPI
>> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
>> with their incredible documentation (documents/acpica-reference.pdf). It
>> would be great if BITS' ACPI module had a list of commands, to see what
>> is there to play with.
> 
> We don't come close to the level of documentation for ACPICA, but we do
> have pydoc documentation for the modules in BITS, including acpi.  You
> can run help("acpi") within BITS, or read acpi.py.  We've tried to make
> sure all of the methods considered "API" have docstrings.

I'll have to digest this some, and play with it.

>> (3) I should mention that QEMU generates ACPI also for arm/aarch64
>> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same edk2
>> module as noted above (built for arm/aarch64) does the guest side
>> processing. Do you think it's possible to use BITS in arm/aarch64 VMs?
> 
> Some folks from Linaro started looking into a BITS arm64 port.

Great!

>> ... I apologize if tools / documentation already exist for this kind of
>> development work; everyone please educate me then. I hope my questions
>> make at least some sense; I realize this email isn't well organized.
> 
> Makes perfect sense, and thanks for your mail!  I love the idea of using
> BITS to test qemu's own ACPI.

Thank you very much! :)

(I must say, I have found the LWN article at just the right time. I
intend to start implementing a VMGenID device for QEMU, and it's all
ACPI based. Here's our design for that:
<http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
already dreading the need for a Linux guest driver, in order to
white-box test the device & the ACPI stuff from the guest side. :))

Cheers!
Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Moore, Robert]

> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Thursday, September 03, 2015 8:54 AM
> To: Josh Triplett
> Cc: edk2-devel-01; qemu devel list; Michael Tsirkin; Igor Mammedov; Marcel
> Apfelbaum; Paolo Bonzini; Shannon Zhao; Moore, Robert; Smith, Jonathan D;
> Jake Edge
> Subject: Re: "Using Python to investigate EFI and ACPI"
> 
> On 09/03/15 16:50, Josh Triplett wrote:
> > On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> >> Hi Josh,
> >>
> >> I just found the LWN.net article with the title in the subject:
> >>
> >>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> >>
> >> ("Thank you LWN", of course. :))
> >>
> >> This got me kind of excited. In your presentation, according to the
> >> article, you were using KVM and OVMF to showcase the ACPI features of
> >> "BITS". Further, in
> >>
> >>   http://biosbits.org/scripting/
> >
> > (Note: that page is badly out of date, by a couple of years, and we
> > need to drop it in favor of the various more recent presentations.
> > All of that functionality still exists, but we have an interactive
> > Python REPL now, and the ACPI methods have moved to the "acpi" module
> > and become easier to use.)
> >
> >> you write, "You can also find arbitrary ACPI objects by path, and
> >> evaluate arbitrary ACPI methods, with the return value decoded into a
> >> Python object".
> >
> > Right.  In particular, you can call acpi.get_objpaths to list the ACPI
> > tree, and acpi.evaluate to evaluate an arbitrary method.
> >
> >> I'm excited because for virtualization we might have the reverse use
> >> case: use BITS to test / debug / develop the ACPI generator of QEMU.
> >> :)
> >
> > Awesome.  Unit testing and CI for QEMU sounds like an ideal use case
> > for BITS.
> >
> >> As you may now, QEMU generates almost all of its ACPI tables
> >> dynamically, providing the guest with an "ACPI payload" that
> >> describes the actual virtual hardware as accurately as possible.
> >
> > Many BIOSes do that, since the hardware configuration may vary at
> > runtime.
> >
> >> Then this payload is passed to the guest firmware (SeaBIOS or OVMF)
> >> over "fw_cfg" (which is a simple protocol, comprising, at this point,
> >> one selector and one data register, which are IO ports or MMIO
> >> locations -- see "docs/specs/fw_cfg.txt" in QEMU and
> >> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).
> >
> > Interesting; I hadn't seen that protocol before.
> >
> > Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> > does the last address written to 0x510 get saved system-wide, making
> > it unsafe for concurrent access?
> 
> I think fw_cfg is not meant to be accessed by several CPUs concurrently.
> The protocol is stateful (selected key, offset within blob associated with
> selected key, etc), and "accessing CPU" is not part of that state.
> 
> >
> >> The guest firmware downloads the "ACPI payload" over fw_cfg, does
> >> some generic processing on them (we call this the "ACPI linker/loader
> >> script"), and ultimately installs the ACPI tables in a way that is
> >> appropriate for the firmware in question.
> >>
> >> To be more concrete, the tables you investigated in your presentation
> >> with BITS were generated (very roughly) by the following files in QEMU:
> >>
> >>   hw/acpi/aml-build.c
> >>   hw/acpi/bios-linker-loader.c
> >>   hw/acpi/...
> >>   hw/i386/acpi-build.c
> >>
> >> and processed and installed by the following source file in edk2:
> >>
> >>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
> >>
> >> With this background, you can probably see where I'm going with this.
> >> It is not really easy to *test* the AML methods that QEMU generates
> >> (piecing them together, with helper functions, from AML primitives),
> >> without dedicated guest kernel drivers. I think the only method that
> >> I know of is the following:
> >>
> >> - in the Linux guest, dump the ACPI tables with acpidump, or save
> >> them from sysfs directly (/sys/firmware/acpi/tables)
> >> - pass the DSDT and the SSDTs (and any data tables referenced by
> >> them?) to AcpiExec from the ACPICA suite
> >> - work with AcpiExec
> >>
> >> But, for simple testing, can we maybe run your tool within the guest,
> >> before the runtime OS boots?
> >
> > Yes, absolutely.  We have a batch-mode testing mechanism based on a
> > config file; you'd probably want to make use of that.  With some
> > extensions, it could dump results either to an emulated serial port or
> > some other interface that you can read from outside qemu.  We also
> > need to work on making the results more machine-parseable for
> automation.
> 
> While I certainly don't discount automation, my primary use case is
> interactive development / testing. :) (Although, I can see myself canning
> some commands in a script or config file, and invoking *that*
> interactively. Which was your point, probably.)
> 
> >
> >> Although AcpiDump is being ported to edk2:
> >>
> >>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
> >>
> >> I'm unsure if it will be able to evaluate / execute AML methods. (If
> >> it parallels the "acpidump" utility we have on GNU/Linux distros, it
> >> won't.)
> >
> > Probably not.  And I haven't heard of any plans to port acpiexec to
> > EDK2.
> >
> >> Thus it would be awesome if we had some AcpiExec-like functionality
> >> early on in the guest (for example in the form of a UEFI Shell
> >> application, or as a python tool that runs within the edk2 Python
> >> port, or even in grub).
> >>
> >> For example, assume your runtime guest OS is Windows (with its picky,
> >> closed-source ACPI interpreter); you make a change in QEMU's ACPI
> >> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
> >> verify the change "by eye", exit the shell, and *then* continue
> >> booting Windows. (Which will hopefully not BSOD at that point, after
> >> the verification with BITS / AcpiExec etc.)
> >>
> >> So, I have three questions:
> >>
> >> (1) What is the relationship between the ACPI facility of BITS, and
> ACPICA?
> >
> > BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA
> > as a git submodule and build it as part of BITS.  acpi.evaluate uses
> > the ACPICA interpreter.
> 
> Awesome! :)
> 
> Another question: when you execute an AML method that does, say, IO port
> access, does the AML interpreter of ACPICA actually *perform* that IO port
> access? Because, the one that is embedded in Linux obviously does, and the
> one that is embedded in the userspace ACPICA command line utility
> "acpiexec" obviously doesn't.

We have recently integrated the ACPICA AML debugger with the Linux kernel,
and we have a userspace utility that allows communication with the debugger.
This allows the user to enter debugger commands such as "execute control method",
and the commands are carried out by the kernel code -- thus talking to the
actual hardware.

Bob

> 
> I assume (and very much hope) that the IO port access *is* performed from
> BITS, simply because you developed it for physical machines, and it
> wouldn't make much sense to avoid actual hardware access that was
> implemented by the BIOS vendor for that platform.
> 
> If that is the case, then this tool could become the killer ACPI tester
> for QEMU developers -- the hardware accesses in the AML methods generated
> by QEMU would actually poke QEMU devices! (Unlike the userspace "acpiexec"
> utility.) It would completely detach Linux guest driver development from
> host side / firmware development. \o/
> 
> >
> >> (2) Is there a bit more comprehensive documentation about the ACPI
> >> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged
> >> me with their incredible documentation
> >> (documents/acpica-reference.pdf). It would be great if BITS' ACPI
> >> module had a list of commands, to see what is there to play with.
> >
> > We don't come close to the level of documentation for ACPICA, but we
> > do have pydoc documentation for the modules in BITS, including acpi.
> > You can run help("acpi") within BITS, or read acpi.py.  We've tried to
> > make sure all of the methods considered "API" have docstrings.
> 
> I'll have to digest this some, and play with it.
> 
> >> (3) I should mention that QEMU generates ACPI also for arm/aarch64
> >> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same
> >> edk2 module as noted above (built for arm/aarch64) does the guest
> >> side processing. Do you think it's possible to use BITS in arm/aarch64
> VMs?
> >
> > Some folks from Linaro started looking into a BITS arm64 port.
> 
> Great!
> 
> >> ... I apologize if tools / documentation already exist for this kind
> >> of development work; everyone please educate me then. I hope my
> >> questions make at least some sense; I realize this email isn't well
> organized.
> >
> > Makes perfect sense, and thanks for your mail!  I love the idea of
> > using BITS to test qemu's own ACPI.
> 
> Thank you very much! :)
> 
> (I must say, I have found the LWN article at just the right time. I intend
> to start implementing a VMGenID device for QEMU, and it's all ACPI based.
> Here's our design for that:
> <http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
> already dreading the need for a Linux guest driver, in order to white-box
> test the device & the ACPI stuff from the guest side. :))
> 
> Cheers!
> Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

On 09/03/15 17:57, Moore, Robert wrote:
> 
> 
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Thursday, September 03, 2015 8:54 AM
>> To: Josh Triplett
>> Cc: edk2-devel-01; qemu devel list; Michael Tsirkin; Igor Mammedov; Marcel
>> Apfelbaum; Paolo Bonzini; Shannon Zhao; Moore, Robert; Smith, Jonathan D;
>> Jake Edge
>> Subject: Re: "Using Python to investigate EFI and ACPI"
>>
>> Another question: when you execute an AML method that does, say, IO port
>> access, does the AML interpreter of ACPICA actually *perform* that IO port
>> access? Because, the one that is embedded in Linux obviously does, and the
>> one that is embedded in the userspace ACPICA command line utility
>> "acpiexec" obviously doesn't.
> 
> We have recently integrated the ACPICA AML debugger with the Linux kernel,
> and we have a userspace utility that allows communication with the debugger.
> This allows the user to enter debugger commands such as "execute control method",
> and the commands are carried out by the kernel code -- thus talking to the
> actual hardware.

... Was it merged in commit c91c5b276b? The series at
<https://lkml.org/lkml/2015/8/24/921>? Or is that "only" a followup
series to the main work?

Can you please provide a pointer to the userspace tool as well?

This thread is getting better and better. :)

Thank you!
Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Moore, Robert]

> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Thursday, September 03, 2015 9:17 AM
> To: Moore, Robert; Josh Triplett
> Cc: edk2-devel-01; qemu devel list; Michael Tsirkin; Igor Mammedov; Marcel
> Apfelbaum; Paolo Bonzini; Shannon Zhao; Smith, Jonathan D; Jake Edge
> Subject: Re: "Using Python to investigate EFI and ACPI"
> 
> On 09/03/15 17:57, Moore, Robert wrote:
> >
> >
> >> -----Original Message-----
> >> From: Laszlo Ersek [mailto:lersek@redhat.com]
> >> Sent: Thursday, September 03, 2015 8:54 AM
> >> To: Josh Triplett
> >> Cc: edk2-devel-01; qemu devel list; Michael Tsirkin; Igor Mammedov;
> >> Marcel Apfelbaum; Paolo Bonzini; Shannon Zhao; Moore, Robert; Smith,
> >> Jonathan D; Jake Edge
> >> Subject: Re: "Using Python to investigate EFI and ACPI"
> >>
> >> Another question: when you execute an AML method that does, say, IO
> >> port access, does the AML interpreter of ACPICA actually *perform*
> >> that IO port access? Because, the one that is embedded in Linux
> >> obviously does, and the one that is embedded in the userspace ACPICA
> >> command line utility "acpiexec" obviously doesn't.
> >
> > We have recently integrated the ACPICA AML debugger with the Linux
> > kernel, and we have a userspace utility that allows communication with
> the debugger.
> > This allows the user to enter debugger commands such as "execute
> > control method", and the commands are carried out by the kernel code
> > -- thus talking to the actual hardware.
> 
> ... Was it merged in commit c91c5b276b? The series at
> <https://lkml.org/lkml/2015/8/24/921>? Or is that "only" a followup series
> to the main work?
> 
> Can you please provide a pointer to the userspace tool as well?

We've just got it working in the last few days. There are still a few issues
to figure out, and we hope to release it in the next few weeks.

Bob



> 
> This thread is getting better and better. :)
> 
> Thank you!
> Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[josh]

On Thu, Sep 03, 2015 at 05:53:45PM +0200, Laszlo Ersek wrote:
> On 09/03/15 16:50, Josh Triplett wrote:
> > On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> >> Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
> >> "fw_cfg" (which is a simple protocol, comprising, at this point, one
> >> selector and one data register, which are IO ports or MMIO locations --
> >> see "docs/specs/fw_cfg.txt" in QEMU and
> >> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).
> > 
> > Interesting; I hadn't seen that protocol before.
> > 
> > Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> > does the last address written to 0x510 get saved system-wide, making it
> > unsafe for concurrent access?
> 
> I think fw_cfg is not meant to be accessed by several CPUs concurrently.
> The protocol is stateful (selected key, offset within blob associated
> with selected key, etc), and "accessing CPU" is not part of that state.

Not that hard to fix; just keep all the state in the accessing CPU
rather than the system.  Current processors do that for the PCI I/O port
pair, to avoid race conditions.  You could easily do that for the fw_cfg
I/O ports.  As a bonus, you then wouldn't need to take any kind of lock
around accesses to that state, because the CPU owns that state.

(That's the easy fix; the harder fix would be creating a new race-free
MMIO protocol and mapping all of the data structures into memory
directly, which would provide a performance benefit as well.  I'd love
to see a general version of such a protocol for a more efficient virtio
filesystem, though in the simpler case of fw_cfg you can just map all of
the structures into memory.)

> >> With this background, you can probably see where I'm going with this. It
> >> is not really easy to *test* the AML methods that QEMU generates
> >> (piecing them together, with helper functions, from AML primitives),
> >> without dedicated guest kernel drivers. I think the only method that I
> >> know of is the following:
> >>
> >> - in the Linux guest, dump the ACPI tables with acpidump, or save them
> >> from sysfs directly (/sys/firmware/acpi/tables)
> >> - pass the DSDT and the SSDTs (and any data tables referenced by them?)
> >> to AcpiExec from the ACPICA suite
> >> - work with AcpiExec
> >>
> >> But, for simple testing, can we maybe run your tool within the guest,
> >> before the runtime OS boots?
> > 
> > Yes, absolutely.  We have a batch-mode testing mechanism based on a
> > config file; you'd probably want to make use of that.  With some
> > extensions, it could dump results either to an emulated serial port or
> > some other interface that you can read from outside qemu.  We also need
> > to work on making the results more machine-parseable for automation.
> 
> While I certainly don't discount automation, my primary use case is
> interactive development / testing. :) (Although, I can see myself
> canning some commands in a script or config file, and invoking *that*
> interactively. Which was your point, probably.)

Interactive development and testing is even easier; you can do that
today.  And yes, if you find yourself doing the same thing repeatedly,
you should put it in a module you can run.  Turning the results into an
automated regression test would be even better. :)

> >> Thus it would be awesome if we had some AcpiExec-like functionality
> >> early on in the guest (for example in the form of a UEFI Shell
> >> application, or as a python tool that runs within the edk2 Python port,
> >> or even in grub).
> >>
> >> For example, assume your runtime guest OS is Windows (with its picky,
> >> closed-source ACPI interpreter); you make a change in QEMU's ACPI
> >> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
> >> verify the change "by eye", exit the shell, and *then* continue booting
> >> Windows. (Which will hopefully not BSOD at that point, after the
> >> verification with BITS / AcpiExec etc.)
> >>
> >> So, I have three questions:
> >>
> >> (1) What is the relationship between the ACPI facility of BITS, and ACPICA?
> > 
> > BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA as
> > a git submodule and build it as part of BITS.  acpi.evaluate uses the
> > ACPICA interpreter.
> 
> Awesome! :)
> 
> Another question: when you execute an AML method that does, say, IO port
> access, does the AML interpreter of ACPICA actually *perform* that IO
> port access? Because, the one that is embedded in Linux obviously does,
> and the one that is embedded in the userspace ACPICA command line
> utility "acpiexec" obviously doesn't.

You need to pass unsafe_io=True to evaluate, in which case it'll do I/O.
Otherwise, it'll ignore I/O.  (On our TODO list: ignoring but logging
I/O so we can look at the I/O accesses as part of the test.)

Actually, that reminds me: we should probably fix AcpiOsWriteMemory to
do the same thing.

> I assume (and very much hope) that the IO port access *is* performed
> from BITS, simply because you developed it for physical machines, and it
> wouldn't make much sense to avoid actual hardware access that was
> implemented by the BIOS vendor for that platform.

We want to default to not performing those accesses, but we definitely
have the option to do so if you know you *want* to trigger real I/O.

> If that is the case, then this tool could become the killer ACPI tester
> for QEMU developers -- the hardware accesses in the AML methods
> generated by QEMU would actually poke QEMU devices! (Unlike the
> userspace "acpiexec" utility.) It would completely detach Linux guest
> driver development from host side / firmware development. \o/

That's exactly why we went with a pre-OS environment rather than an OS;
you don't want to undermine the OS, and you don't want your tests
affected by whatever the OS has done.

> >> (2) Is there a bit more comprehensive documentation about the ACPI
> >> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
> >> with their incredible documentation (documents/acpica-reference.pdf). It
> >> would be great if BITS' ACPI module had a list of commands, to see what
> >> is there to play with.
> > 
> > We don't come close to the level of documentation for ACPICA, but we do
> > have pydoc documentation for the modules in BITS, including acpi.  You
> > can run help("acpi") within BITS, or read acpi.py.  We've tried to make
> > sure all of the methods considered "API" have docstrings.
> 
> I'll have to digest this some, and play with it.

Please feel free to ask if you have any questions.  Which reminds me, I
still need to get a BITS mailing list set up.

> >> ... I apologize if tools / documentation already exist for this kind of
> >> development work; everyone please educate me then. I hope my questions
> >> make at least some sense; I realize this email isn't well organized.
> > 
> > Makes perfect sense, and thanks for your mail!  I love the idea of using
> > BITS to test qemu's own ACPI.
> 
> Thank you very much! :)
> 
> (I must say, I have found the LWN article at just the right time. I
> intend to start implementing a VMGenID device for QEMU, and it's all
> ACPI based. Here's our design for that:
> <http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
> already dreading the need for a Linux guest driver, in order to
> white-box test the device & the ACPI stuff from the guest side. :))

Interesting!  Yeah, BITS should make testing that trivial.  You can read
out the identifier, snapshot and resume, and read it out again.

One request there: please make that device optional in qemu, because
some users of qemu and snapshots specifically *won't* want the OS to
know that anything has happened.

- Josh Triplett

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

On 09/03/15 18:41, josh@joshtriplett.org wrote:
> On Thu, Sep 03, 2015 at 05:53:45PM +0200, Laszlo Ersek wrote:
>> On 09/03/15 16:50, Josh Triplett wrote:

>>> Do you virtualize those I/O ports by CPU, to make them thread-safe, or
>>> does the last address written to 0x510 get saved system-wide, making it
>>> unsafe for concurrent access?
>>
>> I think fw_cfg is not meant to be accessed by several CPUs concurrently.
>> The protocol is stateful (selected key, offset within blob associated
>> with selected key, etc), and "accessing CPU" is not part of that state.
> 
> Not that hard to fix; just keep all the state in the accessing CPU
> rather than the system.  Current processors do that for the PCI I/O port
> pair, to avoid race conditions.  You could easily do that for the fw_cfg
> I/O ports.  As a bonus, you then wouldn't need to take any kind of lock
> around accesses to that state, because the CPU owns that state.
> 
> (That's the easy fix; the harder fix would be creating a new race-free
> MMIO protocol and mapping all of the data structures into memory
> directly, which would provide a performance benefit as well.  I'd love
> to see a general version of such a protocol for a more efficient virtio
> filesystem, though in the simpler case of fw_cfg you can just map all of
> the structures into memory.)

Well, first of all it should be justified why concurrent access to
fw_cfg would be necessary / useful :) But, I'm not arguing against it
per se; until very-very recently, the only client looking at fw_cfg has
been the firmware (it's called "firmware config" after all), and it's
quite natural to require that part of the firmware to be single-threaded.

Now, there's a fully independent discussion / development underway that
might want to expose fw_cfg to the kernel too. Let's not get into that
here, please talk to Gabriel for references :) (Cc'd).

... If you feel tempted to ask "why are then fw_cfg bindings for ARM
documented in the kernel?": because when we were extending fw_cfg from
x86 to ARM, advertizing the affected register block in the DT that QEMU
generated became a necessity, and Peter Maydell (QEMU maintainer and ARM
guru) proposed that I document the fw_cfg-related DT stuff *in the
kernel Documentation tree* -- simply because that was the largest, most
central registry of such DT bindings. This is my recollection of it anyway.

There's *yet another* development underway for speeding up fw_cfg
transfers; please talk to Marc Marí (Cc'd) about those :)

In any case, if what you need resembles a "general virtio filesystem",
then please just use that -- a virtio-block or virtio-scsi disk, with a
normal filesystem on it. The protocol is industry standard and the
performance of the QEMU (and kernel) implementation is splendid.

(For ad-hoc uses, even the "vfat" backend can be used, which practically
implements semihosting.)

fw_cfg was always meant as a set of small bits for the firmware to
consume. The fact that we squeeze kernel and initrd blobs through it is
just historical abuse that is now impossible to eradicate (and must be
sped up instead :(). Whereas other creative reasons that Gabriel could
tell you about why the kernel could be interested in fw_cfg, are just
that: creativity. Which might be better served by virtio too (this is
the subject of ongoing discussion).

>> Another question: when you execute an AML method that does, say, IO port
>> access, does the AML interpreter of ACPICA actually *perform* that IO
>> port access? Because, the one that is embedded in Linux obviously does,
>> and the one that is embedded in the userspace ACPICA command line
>> utility "acpiexec" obviously doesn't.
> 
> You need to pass unsafe_io=True to evaluate, in which case it'll do I/O.

Fantastic! :)

> Otherwise, it'll ignore I/O.  (On our TODO list: ignoring but logging
> I/O so we can look at the I/O accesses as part of the test.)
> 
> Actually, that reminds me: we should probably fix AcpiOsWriteMemory to
> do the same thing.
> 
>> I assume (and very much hope) that the IO port access *is* performed
>> from BITS, simply because you developed it for physical machines, and it
>> wouldn't make much sense to avoid actual hardware access that was
>> implemented by the BIOS vendor for that platform.
> 
> We want to default to not performing those accesses, but we definitely
> have the option to do so if you know you *want* to trigger real I/O.

Yes, me wants that, badly. :)

>> If that is the case, then this tool could become the killer ACPI tester
>> for QEMU developers -- the hardware accesses in the AML methods
>> generated by QEMU would actually poke QEMU devices! (Unlike the
>> userspace "acpiexec" utility.) It would completely detach Linux guest
>> driver development from host side / firmware development. \o/
> 
> That's exactly why we went with a pre-OS environment rather than an OS;
> you don't want to undermine the OS, and you don't want your tests
> affected by whatever the OS has done.

And, since you implemented it as part of GRUB, it is not tied to either
OVMF or SeaBIOS (speaking in virt terms for now) -- it's flexible.

>> (I must say, I have found the LWN article at just the right time. I
>> intend to start implementing a VMGenID device for QEMU, and it's all
>> ACPI based. Here's our design for that:
>> <http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
>> already dreading the need for a Linux guest driver, in order to
>> white-box test the device & the ACPI stuff from the guest side. :))
> 
> Interesting!  Yeah, BITS should make testing that trivial.  You can read
> out the identifier, snapshot and resume, and read it out again.

That's the plan, yes. :)

> One request there: please make that device optional in qemu, because
> some users of qemu and snapshots specifically *won't* want the OS to
> know that anything has happened.

Oh, absolutely. There's a more direct reason for that, too: if I mess it
up (-> likely) and it slips by reviewers (-> much less likely), it
shouldn't be unleashed on the unsuspecting user base... Plus it is guest
visible (minimally due to the ACPI objects), which could trigger Device
Manager stuff in Windows, etc etc etc.

Thank you!
Laszlo

> - Josh Triplett
> 

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[josh]

On Thu, Sep 03, 2015 at 07:19:40PM +0200, Laszlo Ersek wrote:
> On 09/03/15 18:41, josh@joshtriplett.org wrote:
> > On Thu, Sep 03, 2015 at 05:53:45PM +0200, Laszlo Ersek wrote:
> >> On 09/03/15 16:50, Josh Triplett wrote:
> 
> >>> Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> >>> does the last address written to 0x510 get saved system-wide, making it
> >>> unsafe for concurrent access?
> >>
> >> I think fw_cfg is not meant to be accessed by several CPUs concurrently.
> >> The protocol is stateful (selected key, offset within blob associated
> >> with selected key, etc), and "accessing CPU" is not part of that state.
> > 
> > Not that hard to fix; just keep all the state in the accessing CPU
> > rather than the system.  Current processors do that for the PCI I/O port
> > pair, to avoid race conditions.  You could easily do that for the fw_cfg
> > I/O ports.  As a bonus, you then wouldn't need to take any kind of lock
> > around accesses to that state, because the CPU owns that state.
> > 
> > (That's the easy fix; the harder fix would be creating a new race-free
> > MMIO protocol and mapping all of the data structures into memory
> > directly, which would provide a performance benefit as well.  I'd love
> > to see a general version of such a protocol for a more efficient virtio
> > filesystem, though in the simpler case of fw_cfg you can just map all of
> > the structures into memory.)
> 
> Well, first of all it should be justified why concurrent access to
> fw_cfg would be necessary / useful :) But, I'm not arguing against it
> per se; until very-very recently, the only client looking at fw_cfg has
> been the firmware (it's called "firmware config" after all), and it's
> quite natural to require that part of the firmware to be single-threaded.

As systems get larger, it may start making sense to have multi-threaded
firmware and initialization.

> Now, there's a fully independent discussion / development underway that
> might want to expose fw_cfg to the kernel too. Let's not get into that
> here, please talk to Gabriel for references :) (Cc'd).

That's another reason to want a concurrency-safe protocol. :)

> There's *yet another* development underway for speeding up fw_cfg
> transfers; please talk to Marc Marí (Cc'd) about those :)
> 
> In any case, if what you need resembles a "general virtio filesystem",
> then please just use that -- a virtio-block or virtio-scsi disk, with a
> normal filesystem on it. The protocol is industry standard and the
> performance of the QEMU (and kernel) implementation is splendid.

Not at all what I'm looking for; I'm looking for a *filesystem*, like
virtio-9p, but with significantly better performance.  I agree that
starting from fw_cfg for that is probably a bad idea; it's more that if
a high-performance virtio filesystem existed, it might also work for
fw_cfg. :)

> (For ad-hoc uses, even the "vfat" backend can be used, which practically
> implements semihosting.)

Last time I'd looked at it, that backend didn't support writing, only
reading.  You also couldn't safely change the files with the VM running.
FAT also doesn't support any semblance of POSIX semantics.

I don't think a manufactured block device is the right interface here.

> fw_cfg was always meant as a set of small bits for the firmware to
> consume. The fact that we squeeze kernel and initrd blobs through it is
> just historical abuse that is now impossible to eradicate (and must be
> sped up instead :(). Whereas other creative reasons that Gabriel could
> tell you about why the kernel could be interested in fw_cfg, are just
> that: creativity. Which might be better served by virtio too (this is
> the subject of ongoing discussion).

Sure, any replacement protocol should probably go through virtio. :)

> >> If that is the case, then this tool could become the killer ACPI tester
> >> for QEMU developers -- the hardware accesses in the AML methods
> >> generated by QEMU would actually poke QEMU devices! (Unlike the
> >> userspace "acpiexec" utility.) It would completely detach Linux guest
> >> driver development from host side / firmware development. \o/
> > 
> > That's exactly why we went with a pre-OS environment rather than an OS;
> > you don't want to undermine the OS, and you don't want your tests
> > affected by whatever the OS has done.
> 
> And, since you implemented it as part of GRUB, it is not tied to either
> OVMF or SeaBIOS (speaking in virt terms for now) -- it's flexible.

Right, exactly.  We don't have any plans to make UEFI a hard
requirement, though some future features will likely only work on UEFI
(such as our current WIP support for TCP/IP networking).

- Josh Triplett

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

On 09/03/15 23:25, josh@joshtriplett.org wrote:
> On Thu, Sep 03, 2015 at 07:19:40PM +0200, Laszlo Ersek wrote:

>> In any case, if what you need resembles a "general virtio filesystem",
>> then please just use that -- a virtio-block or virtio-scsi disk, with a
>> normal filesystem on it. The protocol is industry standard and the
>> performance of the QEMU (and kernel) implementation is splendid.
> 
> Not at all what I'm looking for; I'm looking for a *filesystem*, like
> virtio-9p, but with significantly better performance.  I agree that
> starting from fw_cfg for that is probably a bad idea; it's more that if
> a high-performance virtio filesystem existed, it might also work for
> fw_cfg. :)

Thanks for mentioning "virtio-9p", now I remember what to point at
instead of it. I recommend Stefan's slides from this year's KVM forum.

https://kvmforum2015.sched.org/event/bca50b64e0fbea734b855498f25d0753
http://blog.vmsplice.net/2015/08/virtio-vsock-zero-configuration.html

Thanks
Laszlo

Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[josh]

On Fri, Sep 04, 2015 at 01:21:57AM +0200, Laszlo Ersek wrote:
> On 09/03/15 23:25, josh@joshtriplett.org wrote:
> > On Thu, Sep 03, 2015 at 07:19:40PM +0200, Laszlo Ersek wrote:
> 
> >> In any case, if what you need resembles a "general virtio filesystem",
> >> then please just use that -- a virtio-block or virtio-scsi disk, with a
> >> normal filesystem on it. The protocol is industry standard and the
> >> performance of the QEMU (and kernel) implementation is splendid.
> > 
> > Not at all what I'm looking for; I'm looking for a *filesystem*, like
> > virtio-9p, but with significantly better performance.  I agree that
> > starting from fw_cfg for that is probably a bad idea; it's more that if
> > a high-performance virtio filesystem existed, it might also work for
> > fw_cfg. :)
> 
> Thanks for mentioning "virtio-9p", now I remember what to point at
> instead of it. I recommend Stefan's slides from this year's KVM forum.
> 
> https://kvmforum2015.sched.org/event/bca50b64e0fbea734b855498f25d0753
> http://blog.vmsplice.net/2015/08/virtio-vsock-zero-configuration.html

Interesting!  While I'm not sure a network-style protocol is the right
one for a virtual filesystem, vsock certainly has the potential to
significantly improve performance and code cleanliness.

I'd hope, though, that a high-performance virtio filesystem could also
take advantage of the ability to directly mmap a file from outside the
VM into the VM's address space.

In any case, we're getting a bit far afield for the original thread. :)

- Josh Triplett