https booting

https booting

[Gerd Hoffmann]

  Hi,

With the world moving to use https by default people start to ask for
https being enabled by default for the qemu boot roms.

We could simply flip the DOWNLOAD_PROTO_HTTPS switch in
src/config/qemu/general.h (ipxe git repo).  Note that this would only
affect booting in bios mode, for uefi qemu uses the efidrv builds which
implies https support is in the hands of the uefi firmware (edk2/ovmf).

After looking at https://ipxe.org/cfg/crosscert I'm not convinced this
is a good idea though.  This would likely put quite some load to
ca.ipxe.org.  Also that machine becomes a single point of failure for
worldwide ipxe https boot, and looking through the mailing list I've
seen we already had (at least) two outages this year.

What happens if you sent crosscert to the empty string?
Will ipxe fail or will it boot without cert verification?

What does it take to mirror http://ca.ipxe.org/auto/?
Just "wget -r" everything and serve it?

How does edk2 handle the root ca problem?
Other comments?

take care,
  Gerd

Re: https booting

[Daniel P. Berrangé]

On Wed, Jul 22, 2020 at 02:08:27PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> With the world moving to use https by default people start to ask for
> https being enabled by default for the qemu boot roms.
> 
> We could simply flip the DOWNLOAD_PROTO_HTTPS switch in
> src/config/qemu/general.h (ipxe git repo).  Note that this would only
> affect booting in bios mode, for uefi qemu uses the efidrv builds which
> implies https support is in the hands of the uefi firmware (edk2/ovmf).
> 
> After looking at https://ipxe.org/cfg/crosscert I'm not convinced this
> is a good idea though.  This would likely put quite some load to
> ca.ipxe.org.  Also that machine becomes a single point of failure for
> worldwide ipxe https boot, and looking through the mailing list I've
> seen we already had (at least) two outages this year.
> 
> What happens if you sent crosscert to the empty string?
> Will ipxe fail or will it boot without cert verification?
> 
> What does it take to mirror http://ca.ipxe.org/auto/?
> Just "wget -r" everything and serve it?
> 
> How does edk2 handle the root ca problem?

There are two fw_cfg paths

  - etc/edk2/https/ciphers
  - etc/edk2/https/cacerts

The first sets the cipher algorithms that are permitted and their
priority, the second sets the CA certificate bundle.

There's some recently merged code in QEMU to simplify the setup
of the ciphers data via the "tls-cipher-suites" object, but
ultimately libvirt is responsible for passing suitable -fw_cfg
args to QEMU to populate both.

I'd suggest that iPXE needs to support the equivalent kind of
concept, both CA certs, and the cipher priority.

The rationale is that the OS vendor defines CA certs and cipher
priority for the host OS, with optional local administrator
override. Any firmware QEMU runs needs to honour the host OS
settings in this area, so we should have a mechanism for pass
in the relevant data from the host for iPXE IMHO.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [ipxe-devel] https booting

[Michael Brown]

On 22/07/2020 13:08, Gerd Hoffmann wrote:
> With the world moving to use https by default people start to ask for
> https being enabled by default for the qemu boot roms.
> 
> We could simply flip the DOWNLOAD_PROTO_HTTPS switch in
> src/config/qemu/general.h (ipxe git repo).  Note that this would only
> affect booting in bios mode, for uefi qemu uses the efidrv builds which
> implies https support is in the hands of the uefi firmware (edk2/ovmf).

The .efidrv builds can be used for booting via either the UEFI PXE flow 
(using iPXE as just a NIC driver) or via the iPXE flow (using iPXE as an 
application invoked via the NIC driver's EFI_LOAD_FILE_PROTOCOL entry 
point), so it might still be relevant to UEFI as well as BIOS.

> After looking at https://ipxe.org/cfg/crosscert I'm not convinced this
> is a good idea though.  This would likely put quite some load to
> ca.ipxe.org.  Also that machine becomes a single point of failure for
> worldwide ipxe https boot, and looking through the mailing list I've
> seen we already had (at least) two outages this year.

The crosscert fetches are static files (with iPXE including a query 
string only for debugging purposes), and it should be fairly 
straightforward for me to switch to hosting them in AWS S3 or 
equivalent.  The ca.ipxe.org domain is not used for anything else so 
could be pointed at a new hosting infrastructure with no disruption or 
code changes.

I would worry more about the OCSP service for the cross-signed 
certificates, since OCSP does not just serve static responses.  This 
service is currently implemented using openca-ocspd running on a VM in 
AWS.  I'm very open to suggestions on more scalable ways to host this.

> What happens if you sent crosscert to the empty string?

An empty string is equivalent to a deleted setting, so it will fall back 
to using the compiled-in default.

> Will ipxe fail or will it boot without cert verification?

iPXE does need to be able to construct a full certificate chain leading 
back to its trusted root certificate.  If the crosscert source is 
unavailable then otherwise valid certs will be treated as invalid since 
the information required to validate them is not avaiable.

> What does it take to mirror http://ca.ipxe.org/auto/?
> Just "wget -r" everything and serve it?

For the crosscert files, yes.  There's still OCSP to think about; see above.

> How does edk2 handle the root ca problem?

I'm also curious to know!

Thanks,

Michael

Re: [ipxe-devel] https booting

[Michael Brown]

On 22/07/2020 14:21, Michael Brown wrote:
>> After looking at https://ipxe.org/cfg/crosscert I'm not convinced this
>> is a good idea though.  This would likely put quite some load to
>> ca.ipxe.org.  Also that machine becomes a single point of failure for
>> worldwide ipxe https boot, and looking through the mailing list I've
>> seen we already had (at least) two outages this year.
> 
> The crosscert fetches are static files (with iPXE including a query 
> string only for debugging purposes), and it should be fairly 
> straightforward for me to switch to hosting them in AWS S3 or 
> equivalent.  The ca.ipxe.org domain is not used for anything else so 
> could be pointed at a new hosting infrastructure with no disruption or 
> code changes.

I've created https://github.com/ipxe/ipxe/issues/126 to track this part 
of the development work.  Please feel free to review the issue and add 
anything you think might be relevant.

Thanks,

Michael

Re: https booting

[Gerd Hoffmann]

> > How does edk2 handle the root ca problem?
> 
> There are two fw_cfg paths
> 
>   - etc/edk2/https/ciphers
>   - etc/edk2/https/cacerts
> 
> The first sets the cipher algorithms that are permitted and their
> priority, the second sets the CA certificate bundle.

Ok, ipxe should be able to fetch them.  Would be roughly the same as
compiling in the certificates, except that they don't take up space in
the rom and are much easier to update.

What is in cacerts?
Basically /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem of the host
machine?

thanks,
  Gerd

Re: https booting

[Daniel P. Berrangé]

On Wed, Jul 22, 2020 at 03:55:38PM +0200, Gerd Hoffmann wrote:
> > > How does edk2 handle the root ca problem?
> > 
> > There are two fw_cfg paths
> > 
> >   - etc/edk2/https/ciphers
> >   - etc/edk2/https/cacerts
> > 
> > The first sets the cipher algorithms that are permitted and their
> > priority, the second sets the CA certificate bundle.
> 
> Ok, ipxe should be able to fetch them.  Would be roughly the same as
> compiling in the certificates, except that they don't take up space in
> the rom and are much easier to update.



> 
> What is in cacerts?
> Basically /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem of the host
> machine?

Not that file exactly. Instead

   /etc/pki/ca-trust/extracted/edk2/cacerts.bin

which is the same certs, but in a different format:

[quote man update-ca-trust]
       The directory /etc/pki/ca-trust/extracted/edk2/ contains a
       CA certificate bundle ("cacerts.bin") in the "sequence of
       EFI_SIGNATURE_LISTs" format, defined in the UEFI-2.7
       specification, sections "31.4.1 Signature Database" and
       "EFI_CERT_X509_GUID". Distrust information cannot be
       represented in this file format, and distrusted certificates
       are missing from these files. File "cacerts.bin" contains CA
       certificates trusted for TLS server authentication.
[/quote]

On Fedora/RHEL  the "update-ca-trust" tool creates the file in this
format automatically now.

I don't know if that's a useful format for iPXE or not.

We could easily define etc/ipxe/https/{ciphers,cacerts} paths in a
different format if better suited for iPXE. Libvirt can set the right
path depending on whether its booting a VM with EDK2 vs legacy BIOS

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: https booting

[Laszlo Ersek]

On 07/22/20 14:08, Gerd Hoffmann wrote:

> How does edk2 handle the root ca problem?

It has no builtin CA certificate. HTTPS boot will not work until at
least one trusted CA cert is imported.

The setup TUI offers an option to import CA cert(s) from local files
(which must be on such filesystems that edk2 can read).

The platform may set up CA certs without (guest-)user interaction, too.
That's what OVMF and ArmVirtQemu do. On the host side, the command

  p11-kit extract --format=edk2-cacerts --filter=ca-anchors \
    --overwrite --purpose=server-auth <certdb>

translates the host-side trusted CA cert list into a format that edk2
can consume.

This p11-kit command is usually invoked as part of the higher-level command

  update-ca-trust extract

When "p11-kit extract" is invoked like that, then the <certdb> pathname
is (for example)

  /etc/pki/ca-trust/extracted/edk2/cacerts.bin

Then QEMU is launched with the following option:

  -fw_cfg name=etc/edk2/https/cacerts,file=<certdb>

OVMF and ArmVirtQemu then fetch the CA cert list from fw_cfg, and make
the generic TLS code use it:

- 9c7d0d499296 ("OvmfPkg/TlsAuthConfigLib: configure trusted CA certs
for HTTPS boot", 2018-03-30)

- ffe048a0807b ("ArmVirtPkg: handle NETWORK_TLS_ENABLE in ArmVirtQemu*",
2019-06-28)

Thanks
Laszlo

Re: https booting

[Laszlo Ersek]

On 07/22/20 16:13, Daniel P. Berrangé wrote:
> On Wed, Jul 22, 2020 at 03:55:38PM +0200, Gerd Hoffmann wrote:
>>>> How does edk2 handle the root ca problem?
>>>
>>> There are two fw_cfg paths
>>>
>>>   - etc/edk2/https/ciphers
>>>   - etc/edk2/https/cacerts
>>>
>>> The first sets the cipher algorithms that are permitted and their
>>> priority, the second sets the CA certificate bundle.
>>
>> Ok, ipxe should be able to fetch them.  Would be roughly the same as
>> compiling in the certificates, except that they don't take up space in
>> the rom and are much easier to update.
> 
> 
> 
>>
>> What is in cacerts?
>> Basically /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem of the host
>> machine?
> 
> Not that file exactly. Instead
> 
>    /etc/pki/ca-trust/extracted/edk2/cacerts.bin
> 
> which is the same certs, but in a different format:
> 
> [quote man update-ca-trust]
>        The directory /etc/pki/ca-trust/extracted/edk2/ contains a
>        CA certificate bundle ("cacerts.bin") in the "sequence of
>        EFI_SIGNATURE_LISTs" format, defined in the UEFI-2.7
>        specification, sections "31.4.1 Signature Database" and
>        "EFI_CERT_X509_GUID". Distrust information cannot be
>        represented in this file format, and distrusted certificates
>        are missing from these files. File "cacerts.bin" contains CA
>        certificates trusted for TLS server authentication.
> [/quote]
> 
> On Fedora/RHEL  the "update-ca-trust" tool creates the file in this
> format automatically now.
> 
> I don't know if that's a useful format for iPXE or not.
> 
> We could easily define etc/ipxe/https/{ciphers,cacerts} paths in a
> different format if better suited for iPXE.

I agree.

The p11-kit extractor for edk2 was implemented in p11-kit commit range ba6ebb05fc0c..de963b96929b:

  https://github.com/p11-glue/p11-kit/commit/59054e4f9fe3
  https://github.com/p11-glue/p11-kit/commit/ee27f9153a14
  https://github.com/p11-glue/p11-kit/commit/de963b96929b

  https://github.com/p11-glue/p11-kit/pull/137
  https://github.com/p11-glue/p11-kit/pull/139

The dependent "update-ca-trust" changes are here:

  https://src.fedoraproject.org/rpms/ca-certificates/c/6220683f7640?branch=master
  https://src.fedoraproject.org/rpms/ca-certificates/c/34c0da9058d6?branch=master

I think these commits could be used as model for an "iPXE extractor" if necessary.

Thanks,
Laszlo

> Libvirt can set the right
> path depending on whether its booting a VM with EDK2 vs legacy BIOS
> 
> Regards,
> Daniel
> 

Re: [ipxe-devel] https booting

[Michael Brown]

On 22/07/2020 15:13, Daniel P. Berrangé wrote:
> We could easily define etc/ipxe/https/{ciphers,cacerts} paths in a
> different format if better suited for iPXE. Libvirt can set the right
> path depending on whether its booting a VM with EDK2 vs legacy BIOS

The most useful for iPXE would probably be to expose the fw_cfg 
mechanism as a URI scheme.  This would give a general mechanism allowing 
for use cases such as running a script provided by the host via e.g.

   chain fw_cfg:///opt/org.example/script.ipxe

The ${crosscert} setting could then be pointed at a base URL within the 
fw_cfg space, e.g.

   #define CROSSCERT "fw_cfg:///etc/ipxe/crosscert/auto"

This would then work in the same way under either BIOS or UEFI (or other 
custom firmware), would provide a feature with applicability broader 
than just obtaining certificates, and would avoid any potential problems 
from allocating enough RAM to parse every root certificate from iPXE's 
fixed 512kB internal heap.

What do you think?

Michael

Re: [ipxe-devel] https booting

[Gerd Hoffmann]

On Fri, Jul 24, 2020 at 05:19:38PM +0100, Michael Brown wrote:
> On 22/07/2020 15:13, Daniel P. Berrangé wrote:
> > We could easily define etc/ipxe/https/{ciphers,cacerts} paths in a
> > different format if better suited for iPXE. Libvirt can set the right
> > path depending on whether its booting a VM with EDK2 vs legacy BIOS
> 
> The most useful for iPXE would probably be to expose the fw_cfg mechanism as
> a URI scheme.  This would give a general mechanism allowing for use cases
> such as running a script provided by the host via e.g.
> 
>   chain fw_cfg:///opt/org.example/script.ipxe

Looks useful.
(coreboot folks might want add cbfs: support).

> The ${crosscert} setting could then be pointed at a base URL within the
> fw_cfg space, e.g.
> 
>   #define CROSSCERT "fw_cfg:///etc/ipxe/crosscert/auto"
> 
> This would then work in the same way under either BIOS or UEFI (or other
> custom firmware), would provide a feature with applicability broader than
> just obtaining certificates, and would avoid any potential problems from
> allocating enough RAM to parse every root certificate from iPXE's fixed
> 512kB internal heap.

Well, it isn't that easy.

fw_cfg isn't designed for a large number of files, and right now there
is a hard limit of (IIRC) 32 entries.  While adding an entry or two
isn't that much of a problem, adding all *.der files (228 right now)
hosted @ http://ca.ipxe.org/auto/ individually certainly is.

So, we'll have to pass a certificate bundle as single file and have ipxe
pick the certificate it needs from it.

take care,
  Gerd

Re: [ipxe-devel] https booting

[Gerd Hoffmann]

  Hi,

> > After looking at https://ipxe.org/cfg/crosscert I'm not convinced this
> > is a good idea though.  This would likely put quite some load to
> > ca.ipxe.org.  Also that machine becomes a single point of failure for
> > worldwide ipxe https boot, and looking through the mailing list I've
> > seen we already had (at least) two outages this year.
> 
> The crosscert fetches are static files (with iPXE including a query string
> only for debugging purposes), and it should be fairly straightforward for me
> to switch to hosting them in AWS S3 or equivalent.  The ca.ipxe.org domain
> is not used for anything else so could be pointed at a new hosting
> infrastructure with no disruption or code changes.

It's also simple to create a mirror, for example to serve machines
without direct internet connection.

> I would worry more about the OCSP service for the cross-signed certificates,
> since OCSP does not just serve static responses.  This service is currently
> implemented using openca-ocspd running on a VM in AWS.  I'm very open to
> suggestions on more scalable ways to host this.

How much of the crosscert creation process is (or can can be) automated?

Is it an option to update the certificates much more frequently?  Say
generate them every three days, be valid for 7 days (instead of 90)?
Then just don't run a OCSP service?

take care,
  Gerd