All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] grub-shell: Add flexibility in QEMU firmware handling
@ 2023-01-21  6:15 Glenn Washburn
  2023-01-23  9:28 ` Gerd Hoffmann
  0 siblings, 1 reply; 7+ messages in thread
From: Glenn Washburn @ 2023-01-21  6:15 UTC (permalink / raw)
  To: grub-devel, Daniel Kiper; +Cc: Glenn Washburn

The current qemu firmware paths for arm-efi and arm64-efi are not available
on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware
files in the source directory and if not found, look for them in locations
where Debian installs them. Prefer to use the firmware file usable with the
QEMU '-bios' option and if not found use the newer pflash firmware files.
This allows supporting older systems more easily. By looking for files in
the source directory first, system firmware files can be overridden and it
can be ensured that the tests can be run regardless of the distro and where
it stores firmware files. If no firmware files are found, print an error
message telling the user that those files must exist and exit with error.

Do not load the system 32-bit ARM firmware VARS file because it must be
writable to prevent a data exception and boot failure. So in order to use
the VARS file, it must be copied to a writable location, but its quite large
at 64M and is not needed to boot successfully.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 105 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 101 insertions(+), 4 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 75f71dc1a2..7e13960284 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -181,21 +181,92 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	boot=cd
 	console=console
 	trim=1
-	qemuopts="-bios OVMF-ia32.fd $qemuopts"
+	bios=${srcdir}/OVMF32.fd
+	pflash_code=${srcdir}/OVMF32_CODE.fd
+	pflash_vars=${srcdir}/OVMF32_VARS.fd
+	if [ -f "$bios" ]; then
+	    qemuopts="-bios $bios $qemuopts"
+	elif [ -f "$pflash_code" ]; then
+	    qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+	    if [ -f "$pflash_vars" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    fi
+	else
+	    bios=/usr/share/qemu/OVMF32.fd
+	    pflash_code=/usr/share/OVMF/OVMF32_CODE_4M.secboot.fd
+	    pflash_vars=/usr/share/OVMF/OVMF32_VARS_4M.fd
+	    if [ -f "$bios" ]; then
+		qemuopts="-bios $bios $qemuopts"
+	    elif [ -f "$pflash_code" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    else
+		echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2
+		exit 1
+	    fi
+	fi
+	qemuopts="-machine q35,accel=tcg $qemuopts"
 	;;
     x86_64-efi)
 	qemu=qemu-system-x86_64
 	boot=cd
 	console=console
 	trim=1
-	qemuopts="-bios OVMF.fd $qemuopts"
+	bios=${srcdir}/OVMF.fd
+	pflash_code=${srcdir}/OVMF_CODE.fd
+	pflash_vars=${srcdir}/OVMF_VARS.fd
+	if [ -f "$bios" ]; then
+	    qemuopts="-bios $bios $qemuopts"
+	elif [ -f "$pflash_code" ]; then
+	    qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+	    if [ -f "$pflash_vars" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    fi
+	else
+	    bios=/usr/share/qemu/OVMF.fd
+	    pflash_code=/usr/share/OVMF/OVMF_CODE.fd
+	    pflash_vars=/usr/share/OVMF/OVMF_VARS.fd
+	    if [ -f "$bios" ]; then
+		qemuopts="-bios $bios $qemuopts"
+	    elif [ -f "$pflash_code" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    else
+		echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2
+		exit 1
+	    fi
+	fi
 	;;
     arm64-efi)
 	qemu=qemu-system-aarch64
 	boot=hd
 	console=console
 	trim=1
-	qemuopts="-machine virt -cpu cortex-a57 -bios /usr/share/qemu-efi/QEMU_EFI.fd $qemuopts"
+	bios=${srcdir}/AAVMF.fd
+	pflash_code=${srcdir}/AAVMF_CODE.fd
+	pflash_vars=${srcdir}/AAVMF_VARS.fd
+	if [ -f "$bios" ]; then
+	    qemuopts="-bios $bios $qemuopts"
+	elif [ -f "$pflash_code" ]; then
+	    qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+	    if [ -f "$pflash_vars" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    fi
+	else
+	    bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd
+	    pflash_code=/usr/share/AAVMF/AAVMF_CODE.fd
+	    pflash_vars=/usr/share/AAVMF/AAVMF_VARS.fd
+	    if [ -f "$bios" ]; then
+		qemuopts="-bios $bios $qemuopts"
+	    elif [ -f "$pflash_code" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+		qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts"
+	    else
+		echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2
+		exit 1
+	    fi
+	fi
+	qemuopts="-machine virt -cpu cortex-a57 $qemuopts"
 	disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file="
 	serial_port=
 	;;
@@ -204,7 +275,33 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	boot=hd
 	console=console
 	trim=1
-	qemuopts="-machine virt -bios /usr/share/ovmf-arm/QEMU_EFI.fd $qemuopts"
+	bios=${srcdir}/AAVMF32.fd
+	pflash_code=${srcdir}/AAVMF32_CODE.fd
+	pflash_vars=${srcdir}/AAVMF32_VARS.fd
+	if [ -f "$bios" ]; then
+	    qemuopts="-bios $bios $qemuopts"
+	elif [ -f "$pflash_code" ]; then
+	    qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+	    if [ -f "$pflash_vars" ]; then
+		qemuopts="-drive if=pflash,format=raw,unit=1,file=$pflash_vars $qemuopts"
+	    fi
+	else
+	    bios=/usr/share/AAVMF/AAVMF32.fd
+	    pflash_code=/usr/share/AAVMF/AAVMF32_CODE.fd
+	    pflash_vars=/usr/share/AAVMF/AAVMF32_VARS.fd
+	    if [ -f "$bios" ]; then
+		qemuopts="-bios $bios $qemuopts"
+	    elif [ -f "$pflash_code" ]; then
+		# NOTE: Do not use the pflash VARS file because it cannot be
+		# used in readonly. So it would need to be copied to a writable
+		# path, but its large at 64M and not needed for running tests.
+		qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts"
+	    else
+		echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2
+		exit 1
+	    fi
+	fi
+	qemuopts="-machine virt $qemuopts"
 	disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file="
 	serial_port=efi0
 	;;
-- 
2.34.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-21  6:15 [PATCH] grub-shell: Add flexibility in QEMU firmware handling Glenn Washburn
@ 2023-01-23  9:28 ` Gerd Hoffmann
  2023-01-23 20:09   ` Glenn Washburn
  0 siblings, 1 reply; 7+ messages in thread
From: Gerd Hoffmann @ 2023-01-23  9:28 UTC (permalink / raw)
  To: The development of GNU GRUB; +Cc: Daniel Kiper, Glenn Washburn

On Sat, Jan 21, 2023 at 12:15:20AM -0600, Glenn Washburn wrote:
> The current qemu firmware paths for arm-efi and arm64-efi are not available
> on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware
> files in the source directory and if not found, look for them in locations
> where Debian installs them.

I'd suggest to inspect the *.json files in /usr/share/qemu/firmware/ to
find distro-installed firmware files.

> Do not load the system 32-bit ARM firmware VARS file because it must be
> writable to prevent a data exception and boot failure. So in order to use
> the VARS file, it must be copied to a writable location, but its quite large
> at 64M and is not needed to boot successfully.

You can load the VARS file with snapshot=on (and drop readonly=on) to
make things work without copying the file.

take care,
  Gerd



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-23  9:28 ` Gerd Hoffmann
@ 2023-01-23 20:09   ` Glenn Washburn
  2023-01-24  8:16     ` Gerd Hoffmann
  2023-01-24 21:21     ` Robbie Harwood
  0 siblings, 2 replies; 7+ messages in thread
From: Glenn Washburn @ 2023-01-23 20:09 UTC (permalink / raw)
  To: Gerd Hoffmann; +Cc: The development of GNU GRUB, Daniel Kiper

On Mon, 23 Jan 2023 10:28:09 +0100
Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Sat, Jan 21, 2023 at 12:15:20AM -0600, Glenn Washburn wrote:
> > The current qemu firmware paths for arm-efi and arm64-efi are not
> > available on Ubuntu/Debian but are hardcoded. Switch to first
> > looking for firmware files in the source directory and if not
> > found, look for them in locations where Debian installs them.
> 
> I'd suggest to inspect the *.json files in /usr/share/qemu/firmware/
> to find distro-installed firmware files.

Yes, I know about this, but decided against it so as not to do real or
hacked up json parsing and add another dependency (eg. jq). I think
right now the unstated GRUB policy is that these tests are only
officially supported to run on (newer) debian systems, so I felt that
hard coding was reasonable. It occurs to me that its possible (though I
suspect very improbable), that Redhat based distros run the tests when
building the official RPM. Is there a reason that redhat would be very
interested in the change you're suggesting?

Since building GRUB already has a dependency on Python, a tiny script
could be used to extract needed info from the json file. But I still
find that kinda ugly cause the grub-shell script is a shell script. I'm
inclined to say this is good enough as is for now, and if anyone
wants to submit a patch to change it I'm open to endorsing it.

> > Do not load the system 32-bit ARM firmware VARS file because it
> > must be writable to prevent a data exception and boot failure. So
> > in order to use the VARS file, it must be copied to a writable
> > location, but its quite large at 64M and is not needed to boot
> > successfully.
> 
> You can load the VARS file with snapshot=on (and drop readonly=on) to
> make things work without copying the file.

Good to know. Do you know of any benefit to doing this (ie. using the
installed VARS file)?

Thanks for the feedback,
Glenn


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-23 20:09   ` Glenn Washburn
@ 2023-01-24  8:16     ` Gerd Hoffmann
  2023-01-26  5:23       ` Glenn Washburn
  2023-01-24 21:21     ` Robbie Harwood
  1 sibling, 1 reply; 7+ messages in thread
From: Gerd Hoffmann @ 2023-01-24  8:16 UTC (permalink / raw)
  To: Glenn Washburn; +Cc: The development of GNU GRUB, Daniel Kiper

  Hi,

> > > Do not load the system 32-bit ARM firmware VARS file because it
> > > must be writable to prevent a data exception and boot failure. So
> > > in order to use the VARS file, it must be copied to a writable
> > > location, but its quite large at 64M and is not needed to boot
> > > successfully.
> > 
> > You can load the VARS file with snapshot=on (and drop readonly=on) to
> > make things work without copying the file.
> 
> Good to know. Do you know of any benefit to doing this (ie. using the
> installed VARS file)?

Well, the firmware expects it can write to VARS flash.  ovmf has for
historical reasons code to cope with non-flash or readonly VARS (so
-bios works), but armvirt has not.  Which is why you see the failures.

snapshot=on allows the guest write to a in-RAM throwaway snapshot,
which is the same as using a writable temporary copy of the vars
file, but without the need for a temporary file.

take care,
  Gerd



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-23 20:09   ` Glenn Washburn
  2023-01-24  8:16     ` Gerd Hoffmann
@ 2023-01-24 21:21     ` Robbie Harwood
  2023-01-26  4:53       ` Glenn Washburn
  1 sibling, 1 reply; 7+ messages in thread
From: Robbie Harwood @ 2023-01-24 21:21 UTC (permalink / raw)
  To: Glenn Washburn, Gerd Hoffmann; +Cc: The development of GNU GRUB, Daniel Kiper

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

Glenn Washburn <development@efficientek.com> writes:

> On Mon, 23 Jan 2023 10:28:09 +0100
> Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>> On Sat, Jan 21, 2023 at 12:15:20AM -0600, Glenn Washburn wrote:
>> > The current qemu firmware paths for arm-efi and arm64-efi are not
>> > available on Ubuntu/Debian but are hardcoded. Switch to first
>> > looking for firmware files in the source directory and if not
>> > found, look for them in locations where Debian installs them.
>> 
>> I'd suggest to inspect the *.json files in /usr/share/qemu/firmware/
>> to find distro-installed firmware files.
>
> Yes, I know about this, but decided against it so as not to do real or
> hacked up json parsing and add another dependency (eg. jq). I think
> right now the unstated GRUB policy is that these tests are only
> officially supported to run on (newer) debian systems, so I felt that
> hard coding was reasonable. It occurs to me that its possible (though I
> suspect very improbable), that Redhat based distros run the tests when
> building the official RPM. Is there a reason that redhat would be very
> interested in the change you're suggesting?

We don't run the tests during builds.  That has more to do with
buildtimes and historical considerations than the tests themselves.

Personally, I would write this in such a way that we could, e.g., check
the debian locations, and fallback to the fedora locations if they don't
exist, and so forth for other distros.  That said, grub has not been in
favor of handling compatibility in this way in the past and seems to
prefer pushing that burden onto the distros and their users.

Be well,
--Robbie

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-24 21:21     ` Robbie Harwood
@ 2023-01-26  4:53       ` Glenn Washburn
  0 siblings, 0 replies; 7+ messages in thread
From: Glenn Washburn @ 2023-01-26  4:53 UTC (permalink / raw)
  To: Robbie Harwood; +Cc: Gerd Hoffmann, The development of GNU GRUB, Daniel Kiper

On Tue, 24 Jan 2023 16:21:34 -0500
Robbie Harwood <rharwood@redhat.com> wrote:

> Glenn Washburn <development@efficientek.com> writes:
> 
> > On Mon, 23 Jan 2023 10:28:09 +0100
> > Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> >> On Sat, Jan 21, 2023 at 12:15:20AM -0600, Glenn Washburn wrote:
> >> > The current qemu firmware paths for arm-efi and arm64-efi are not
> >> > available on Ubuntu/Debian but are hardcoded. Switch to first
> >> > looking for firmware files in the source directory and if not
> >> > found, look for them in locations where Debian installs them.
> >> 
> >> I'd suggest to inspect the *.json files in
> >> /usr/share/qemu/firmware/ to find distro-installed firmware files.
> >
> > Yes, I know about this, but decided against it so as not to do real
> > or hacked up json parsing and add another dependency (eg. jq). I
> > think right now the unstated GRUB policy is that these tests are
> > only officially supported to run on (newer) debian systems, so I
> > felt that hard coding was reasonable. It occurs to me that its
> > possible (though I suspect very improbable), that Redhat based
> > distros run the tests when building the official RPM. Is there a
> > reason that redhat would be very interested in the change you're
> > suggesting?
> 
> We don't run the tests during builds.  That has more to do with
> buildtimes and historical considerations than the tests themselves.

I don't understand the last sentence, but I'm gathering that you guys
don't ever run these tests.

> Personally, I would write this in such a way that we could, e.g.,
> check the debian locations, and fallback to the fedora locations if
> they don't exist, and so forth for other distros.  That said, grub
> has not been in favor of handling compatibility in this way in the
> past and seems to prefer pushing that burden onto the distros and
> their users.

Yeah, I guess I don't personally care enough, even if its trivial. Is
the point that it would make it easier for someone (not me) to send a
patch adding Redhat support? If so, then they could add what your
proposing in that patch. If other distros want to patch their source so
that users getting the source ball can run the tests without having to
manually change the paths, this current implementation makes that easy,
just change the paths in a patch. I'm not completely opposed to the
idea if someone else wants to do it, but I'm not personally interested
in making it uglier and more complex than it already is. Despite this,
I appreciate your input.

Glenn


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] grub-shell: Add flexibility in QEMU firmware handling
  2023-01-24  8:16     ` Gerd Hoffmann
@ 2023-01-26  5:23       ` Glenn Washburn
  0 siblings, 0 replies; 7+ messages in thread
From: Glenn Washburn @ 2023-01-26  5:23 UTC (permalink / raw)
  To: Gerd Hoffmann; +Cc: The development of GNU GRUB, Daniel Kiper

On Tue, 24 Jan 2023 09:16:26 +0100
Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
> 
> > > > Do not load the system 32-bit ARM firmware VARS file because it
> > > > must be writable to prevent a data exception and boot failure.
> > > > So in order to use the VARS file, it must be copied to a
> > > > writable location, but its quite large at 64M and is not needed
> > > > to boot successfully.
> > > 
> > > You can load the VARS file with snapshot=on (and drop
> > > readonly=on) to make things work without copying the file.
> > 
> > Good to know. Do you know of any benefit to doing this (ie. using
> > the installed VARS file)?
> 
> Well, the firmware expects it can write to VARS flash.  ovmf has for
> historical reasons code to cope with non-flash or readonly VARS (so
> -bios works), but armvirt has not.  Which is why you see the failures.

Well armvirt seems to cope fine if there's no VARS flash provided, it
just dies when there is one provided which is readonly.

> snapshot=on allows the guest write to a in-RAM throwaway snapshot,
> which is the same as using a writable temporary copy of the vars
> file, but without the need for a temporary file.

Yeah, I get that, but armvirt doesn't even need the VARS file to run.
You'll see in this patch that for 32-bit arm, no VARS flash is created
when loading the firmware code from the system path or when loading the
firmware code from the grub source directory and the VARS file doesn't
exist. So what benefit is there to providing the system VARS as a flash
device? I can't think of a benefit to using the VARS file if the tests
all pass fine without it.

I'd prefer to have as few dependencies on files not in the GRUB source
as possible. So I like that I don't need to use the system VARS file on
32-bit arm. I wish I could do the same on the other platforms.

This patch does no temporary copying of VARS files on any platform and
all the QEMU tests run as expected. So using "snapshot=on" won't
prevent any copying, which is the main benefit that I see. I do
appreciate this suggestion as it might be useful in other work with
QEMU, I just don't see how its beneficial here.

Glenn


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-01-26  5:24 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-21  6:15 [PATCH] grub-shell: Add flexibility in QEMU firmware handling Glenn Washburn
2023-01-23  9:28 ` Gerd Hoffmann
2023-01-23 20:09   ` Glenn Washburn
2023-01-24  8:16     ` Gerd Hoffmann
2023-01-26  5:23       ` Glenn Washburn
2023-01-24 21:21     ` Robbie Harwood
2023-01-26  4:53       ` Glenn Washburn

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.