Can't find a solution to a failed secure boot kernel loading

Can't find a solution to a failed secure boot kernel loading

[Łukasz Piątkowski]

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

Hi everyone - I'm new here!

Sorry for going with my problem directly to the grub-devel maling list, but
I'm pretty sure my problem is GRUB related. Still, I've spent some hours
trying to find a solution on the Internet and I failed :( So, here it comes
- if anyone has time to explain my problem to a layman, it would be
awesome. Even better, if you can maybe answer here on stackoverflow, where
it can be easier to find, I believe (
https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature
).

I'm running ubuntu with Secure Boot on. Everything works fine when I use a
kernel that comes packaged from cannonical. Still, I have issues running a
self-signed kernel (this is actually an externally built kernel, that I
have verified and want to use for my own machine). I'm pretty sure my
signature with MOK key is OK (verification below), but still when I try to
boot the kernel from grub, after selecting the correct entry, I get an
error that reads "Loading ... error: bad shim signature." I'm wrapping my
head around it and can't find a solution. Why, even though both kernels are
signed with MOK keys, one of them works and the other doesn't?

Here's info about kernel signatures:

root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert
/var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
Image was already signed; adding additional signature

root@T495:~# sbverify --list /boot/vmlinuz
signature 1
image signature issuers:
 - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot Signing/emailAddress=
example@example.com
image signature certificates:
 - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=
example@example.com
   issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=
example@example.com
signature 2
image signature issuers:
 - /CN=ubuntu Secure Boot Module Signature key
image signature certificates:
 - subject: /CN=ubuntu Secure Boot Module Signature key
   issuer:  /CN=ubuntu Secure Boot Module Signature key


And here about MOK keys:

root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem -fingerprint
-noout
SHA1 Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
SHA1 Fingerprint:
81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7

If there are any docs that help understand that, I'm happy to be redirected
there :)

piontec

[-- Attachment #2: Type: text/html, Size: 2933 bytes --]

Re: Can't find a solution to a failed secure boot kernel loading

[James Bottomley]

On Tue, 2022-05-10 at 12:28 +0200, Łukasz Piątkowski wrote:
> Hi everyone - I'm new here!
> 
> Sorry for going with my problem directly to the grub-devel maling
> list, but I'm pretty sure my problem is GRUB related. Still, I've
> spent some hours trying to find a solution on the Internet and I
> failed :( So, here it comes
> - if anyone has time to explain my problem to a layman, it would be
> awesome. Even better, if you can maybe answer here on stackoverflow,
> where it can be easier to find, I believe (
> https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature
> ).
> 
> I'm running ubuntu with Secure Boot on. Everything works fine when I
> use a kernel that comes packaged from cannonical. Still, I have
> issues running a self-signed kernel (this is actually an externally
> built kernel,

Please can you clarify what you're doing, because the sbverify you show
below isn't an externally built kernel, it must be an Ubuntu kernel to
have an ubuntu signature on it, or is that ubuntu certificate one you
created?

Is your problem that this kernel with the dual signature won't boot
even though the Ubuntu key should be in shim/mok, or that you have
another kernel you didn't attach the sbverify output for that won't
boot?

James

>  that I have verified and want to use for my own machine). I'm pretty
> sure my signature with MOK key is OK (verification below), but still
> when I try to boot the kernel from grub, after selecting the correct
> entry, I get an error that reads "Loading ... error: bad shim
> signature." I'm wrapping my head around it and can't find a solution.
> Why, even though both kernels are signed with MOK keys, one of them
> works and the other doesn't?
> 
> Here's info about kernel signatures:
> 
> root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert
> /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
> Image was already signed; adding additional signature
> 
> root@T495:~# sbverify --list /boot/vmlinuz
> signature 1
> image signature issuers:
>  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot
> Signing/emailAddress=
> example@example.com
> image signature certificates:
>  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=
> example@example.com
>    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=
> example@example.com
> signature 2
> image signature issuers:
>  - /CN=ubuntu Secure Boot Module Signature key
> image signature certificates:
>  - subject: /CN=ubuntu Secure Boot Module Signature key
>    issuer:  /CN=ubuntu Secure Boot Module Signature key
> 
> 
> And here about MOK keys:
> 
> root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem
> -fingerprint
> -noout
> SHA1
> Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:
> F7
> root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
> SHA1 Fingerprint:
> 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
> 
> If there are any docs that help understand that, I'm happy to be
> redirected
> there :)

Re: Can't find a solution to a failed secure boot kernel loading

[Dimitri John Ledkov]

the MOK key as generated by Ubuntu/Debian tooling, creates a signing
certificate that self-limits itself to only support Kernel Module
signing.
Signatures made by such certificate, are not trusted by shim for the
purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
linux).
I also responded this on stackoverflow.

The automatically generated MOK key is only usable to sign kernel
modules, i.e. self-built DKMS modules.

-- 
okurrr,

Dimitri

On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com> wrote:
>
> Hi everyone - I'm new here!
>
> Sorry for going with my problem directly to the grub-devel maling list, but I'm pretty sure my problem is GRUB related. Still, I've spent some hours trying to find a solution on the Internet and I failed :( So, here it comes - if anyone has time to explain my problem to a layman, it would be awesome. Even better, if you can maybe answer here on stackoverflow, where it can be easier to find, I believe (https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature).
>
> I'm running ubuntu with Secure Boot on. Everything works fine when I use a kernel that comes packaged from cannonical. Still, I have issues running a self-signed kernel (this is actually an externally built kernel, that I have verified and want to use for my own machine). I'm pretty sure my signature with MOK key is OK (verification below), but still when I try to boot the kernel from grub, after selecting the correct entry, I get an error that reads "Loading ... error: bad shim signature." I'm wrapping my head around it and can't find a solution. Why, even though both kernels are signed with MOK keys, one of them works and the other doesn't?
>
> Here's info about kernel signatures:
>
> root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
> Image was already signed; adding additional signature
>
> root@T495:~# sbverify --list /boot/vmlinuz
> signature 1
> image signature issuers:
>  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
> image signature certificates:
>  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
> signature 2
> image signature issuers:
>  - /CN=ubuntu Secure Boot Module Signature key
> image signature certificates:
>  - subject: /CN=ubuntu Secure Boot Module Signature key
>    issuer:  /CN=ubuntu Secure Boot Module Signature key
>
>
> And here about MOK keys:
>
> root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem -fingerprint -noout
> SHA1 Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
> root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
> SHA1 Fingerprint: 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
>
> If there are any docs that help understand that, I'm happy to be redirected there :)
>
> piontec
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: Can't find a solution to a failed secure boot kernel loading

[Łukasz Piątkowski]

[-- Attachment #1: Type: text/plain, Size: 6096 bytes --]

What I'm trying to do is to sign a mainline kernel built by ubuntu (
https://kernel.ubuntu.com/~kernel-ppa/mainline/) with my private key, that
is already enrolled to MOK, and boot it with Secure Boot.

> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
certificate that self-limits itself to only support Kernel Module signing.

OK, that explains why the key in `/var/lib/shim-signed/mok` doesn't work.
Still, I have created my own key as well (listed below for inspection, it
has code signing extension), enrolled that key in MOK and signed the ubuntu
mainline kernel (the kernel I'm trying to boot) with it. The result is
exactly the same. I was using exactly the same procedure a few ubuntu
editions back and it was definitely working. From what I learned so
far, this might be related to the BootHole bug (
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10713) that was
fixed some time ago.

My generated key is:

root@T495:~/mok# openssl x509 -in MOK.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            42:61:86:b2:29:3d:ca:eb:98:87:ae:3d:74:95:c7:f2:63:8f:8a:3b
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure Boot
Signing, emailAddress = example@example.com
        Validity
            Not Before: Feb 18 19:28:16 2020 GMT
            Not After : Jan 25 19:28:16 2120 GMT
        Subject: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure
Boot Signing, emailAddress = example@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus: [cut]
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
            X509v3 Authority Key Identifier:
                EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage:
                Code Signing, 1.3.6.1.4.1.311.10.3.6,
1.3.6.1.4.1.2312.16.1.2
            Netscape Comment:
                OpenSSL Generated Certificate
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value: [cut]

On Tue, May 10, 2022 at 3:26 PM Dimitri John Ledkov <
dimitri.ledkov@canonical.com> wrote:

> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
> certificate that self-limits itself to only support Kernel Module
> signing.
> Signatures made by such certificate, are not trusted by shim for the
> purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
> linux).
> I also responded this on stackoverflow.
>
> The automatically generated MOK key is only usable to sign kernel
> modules, i.e. self-built DKMS modules.
>
> --
> okurrr,
>
> Dimitri
>
> On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com> wrote:
> >
> > Hi everyone - I'm new here!
> >
> > Sorry for going with my problem directly to the grub-devel maling list,
> but I'm pretty sure my problem is GRUB related. Still, I've spent some
> hours trying to find a solution on the Internet and I failed :( So, here it
> comes - if anyone has time to explain my problem to a layman, it would be
> awesome. Even better, if you can maybe answer here on stackoverflow, where
> it can be easier to find, I believe (
> https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature
> ).
> >
> > I'm running ubuntu with Secure Boot on. Everything works fine when I use
> a kernel that comes packaged from cannonical. Still, I have issues running
> a self-signed kernel (this is actually an externally built kernel, that I
> have verified and want to use for my own machine). I'm pretty sure my
> signature with MOK key is OK (verification below), but still when I try to
> boot the kernel from grub, after selecting the correct entry, I get an
> error that reads "Loading ... error: bad shim signature." I'm wrapping my
> head around it and can't find a solution. Why, even though both kernels are
> signed with MOK keys, one of them works and the other doesn't?
> >
> > Here's info about kernel signatures:
> >
> > root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert
> /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
> > Image was already signed; adding additional signature
> >
> > root@T495:~# sbverify --list /boot/vmlinuz
> > signature 1
> > image signature issuers:
> >  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot Signing/emailAddress=
> example@example.com
> > image signature certificates:
> >  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=example@example.com
> >    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=example@example.com
> > signature 2
> > image signature issuers:
> >  - /CN=ubuntu Secure Boot Module Signature key
> > image signature certificates:
> >  - subject: /CN=ubuntu Secure Boot Module Signature key
> >    issuer:  /CN=ubuntu Secure Boot Module Signature key
> >
> >
> > And here about MOK keys:
> >
> > root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem
> -fingerprint -noout
> > SHA1
> Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
> > root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
> > SHA1 Fingerprint:
> 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
> >
> > If there are any docs that help understand that, I'm happy to be
> redirected there :)
> >
> > piontec
> > _______________________________________________
> > Grub-devel mailing list
> > Grub-devel@gnu.org
> > https://lists.gnu.org/mailman/listinfo/grub-devel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 7876 bytes --]

Re: Can't find a solution to a failed secure boot kernel loading

[Dimitri John Ledkov]

On Tue, 10 May 2022 at 15:07, Łukasz Piątkowski <piontec@gmail.com> wrote:
>
> What I'm trying to do is to sign a mainline kernel built by ubuntu (https://kernel.ubuntu.com/~kernel-ppa/mainline/) with my private key, that is already enrolled to MOK, and boot it with Secure Boot.
>
> > the MOK key as generated by Ubuntu/Debian tooling, creates a signing certificate that self-limits itself to only support Kernel Module signing.
>
> OK, that explains why the key in `/var/lib/shim-signed/mok` doesn't work. Still, I have created my own key as well (listed below for inspection, it has code signing extension), enrolled that key in MOK and signed the ubuntu mainline kernel (the kernel I'm trying to boot) with it. The result is exactly the same. I was using exactly the same procedure a few ubuntu editions back and it was definitely working. From what I learned so far, this might be related to the BootHole bug (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10713) that was fixed some time ago.
>
> My generated key is:
>
> root@T495:~/mok# openssl x509 -in MOK.pem -text -noout
> Certificate:
>     Data:
>         Version: 3 (0x2)
>         Serial Number:
>             42:61:86:b2:29:3d:ca:eb:98:87:ae:3d:74:95:c7:f2:63:8f:8a:3b
>         Signature Algorithm: sha256WithRSAEncryption
>         Issuer: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure Boot Signing, emailAddress = example@example.com
>         Validity
>             Not Before: Feb 18 19:28:16 2020 GMT
>             Not After : Jan 25 19:28:16 2120 GMT
>         Subject: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure Boot Signing, emailAddress = example@example.com
>         Subject Public Key Info:
>             Public Key Algorithm: rsaEncryption
>                 Public-Key: (2048 bit)
>                 Modulus: [cut]
>                 Exponent: 65537 (0x10001)
>         X509v3 extensions:
>             X509v3 Subject Key Identifier:
>                 EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>             X509v3 Authority Key Identifier:
>                 EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>             X509v3 Basic Constraints: critical
>                 CA:FALSE
>             X509v3 Extended Key Usage:
>                 Code Signing, 1.3.6.1.4.1.311.10.3.6, 1.3.6.1.4.1.2312.16.1.2

This is bad... certs that have 1.3.6.1.4.1.2312.16.1.2 cannot be used
to sign kernels.

Your cert must _not_ have 1.3.6.1.4.1.2312.16.1.2 EKU set on it.

You cannot use the same certificate to sign both kernel and modules.

>             Netscape Comment:
>                 OpenSSL Generated Certificate
>     Signature Algorithm: sha256WithRSAEncryption
>     Signature Value: [cut]
>
> On Tue, May 10, 2022 at 3:26 PM Dimitri John Ledkov <dimitri.ledkov@canonical.com> wrote:
>>
>> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
>> certificate that self-limits itself to only support Kernel Module
>> signing.
>> Signatures made by such certificate, are not trusted by shim for the
>> purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
>> linux).
>> I also responded this on stackoverflow.
>>
>> The automatically generated MOK key is only usable to sign kernel
>> modules, i.e. self-built DKMS modules.
>>
>> --
>> okurrr,
>>
>> Dimitri
>>
>> On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com> wrote:
>> >
>> > Hi everyone - I'm new here!
>> >
>> > Sorry for going with my problem directly to the grub-devel maling list, but I'm pretty sure my problem is GRUB related. Still, I've spent some hours trying to find a solution on the Internet and I failed :( So, here it comes - if anyone has time to explain my problem to a layman, it would be awesome. Even better, if you can maybe answer here on stackoverflow, where it can be easier to find, I believe (https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature).
>> >
>> > I'm running ubuntu with Secure Boot on. Everything works fine when I use a kernel that comes packaged from cannonical. Still, I have issues running a self-signed kernel (this is actually an externally built kernel, that I have verified and want to use for my own machine). I'm pretty sure my signature with MOK key is OK (verification below), but still when I try to boot the kernel from grub, after selecting the correct entry, I get an error that reads "Loading ... error: bad shim signature." I'm wrapping my head around it and can't find a solution. Why, even though both kernels are signed with MOK keys, one of them works and the other doesn't?
>> >
>> > Here's info about kernel signatures:
>> >
>> > root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
>> > Image was already signed; adding additional signature
>> >
>> > root@T495:~# sbverify --list /boot/vmlinuz
>> > signature 1
>> > image signature issuers:
>> >  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>> > image signature certificates:
>> >  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>> >    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>> > signature 2
>> > image signature issuers:
>> >  - /CN=ubuntu Secure Boot Module Signature key
>> > image signature certificates:
>> >  - subject: /CN=ubuntu Secure Boot Module Signature key
>> >    issuer:  /CN=ubuntu Secure Boot Module Signature key
>> >
>> >
>> > And here about MOK keys:
>> >
>> > root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem -fingerprint -noout
>> > SHA1 Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
>> > root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
>> > SHA1 Fingerprint: 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
>> >
>> > If there are any docs that help understand that, I'm happy to be redirected there :)
>> >
>> > piontec
>> > _______________________________________________
>> > Grub-devel mailing list
>> > Grub-devel@gnu.org
>> > https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: Can't find a solution to a failed secure boot kernel loading

[Łukasz Piątkowski]

[-- Attachment #1: Type: text/plain, Size: 7288 bytes --]

Huh, I've never seen that before... thanks, I'm gonna give it a try and
report back!

On Tue, May 10, 2022 at 4:44 PM Dimitri John Ledkov <
dimitri.ledkov@canonical.com> wrote:

> On Tue, 10 May 2022 at 15:07, Łukasz Piątkowski <piontec@gmail.com> wrote:
> >
> > What I'm trying to do is to sign a mainline kernel built by ubuntu (
> https://kernel.ubuntu.com/~kernel-ppa/mainline/) with my private key,
> that is already enrolled to MOK, and boot it with Secure Boot.
> >
> > > the MOK key as generated by Ubuntu/Debian tooling, creates a signing
> certificate that self-limits itself to only support Kernel Module signing.
> >
> > OK, that explains why the key in `/var/lib/shim-signed/mok` doesn't
> work. Still, I have created my own key as well (listed below for
> inspection, it has code signing extension), enrolled that key in MOK and
> signed the ubuntu mainline kernel (the kernel I'm trying to boot) with it.
> The result is exactly the same. I was using exactly the same procedure a
> few ubuntu editions back and it was definitely working. From what I learned
> so far, this might be related to the BootHole bug (
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10713) that was
> fixed some time ago.
> >
> > My generated key is:
> >
> > root@T495:~/mok# openssl x509 -in MOK.pem -text -noout
> > Certificate:
> >     Data:
> >         Version: 3 (0x2)
> >         Serial Number:
> >             42:61:86:b2:29:3d:ca:eb:98:87:ae:3d:74:95:c7:f2:63:8f:8a:3b
> >         Signature Algorithm: sha256WithRSAEncryption
> >         Issuer: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure
> Boot Signing, emailAddress = example@example.com
> >         Validity
> >             Not Before: Feb 18 19:28:16 2020 GMT
> >             Not After : Jan 25 19:28:16 2120 GMT
> >         Subject: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure
> Boot Signing, emailAddress = example@example.com
> >         Subject Public Key Info:
> >             Public Key Algorithm: rsaEncryption
> >                 Public-Key: (2048 bit)
> >                 Modulus: [cut]
> >                 Exponent: 65537 (0x10001)
> >         X509v3 extensions:
> >             X509v3 Subject Key Identifier:
> >
>  EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
> >             X509v3 Authority Key Identifier:
> >
>  EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
> >             X509v3 Basic Constraints: critical
> >                 CA:FALSE
> >             X509v3 Extended Key Usage:
> >                 Code Signing, 1.3.6.1.4.1.311.10.3.6,
> 1.3.6.1.4.1.2312.16.1.2
>
> This is bad... certs that have 1.3.6.1.4.1.2312.16.1.2 cannot be used
> to sign kernels.
>
> Your cert must _not_ have 1.3.6.1.4.1.2312.16.1.2 EKU set on it.
>
> You cannot use the same certificate to sign both kernel and modules.
>
> >             Netscape Comment:
> >                 OpenSSL Generated Certificate
> >     Signature Algorithm: sha256WithRSAEncryption
> >     Signature Value: [cut]
> >
> > On Tue, May 10, 2022 at 3:26 PM Dimitri John Ledkov <
> dimitri.ledkov@canonical.com> wrote:
> >>
> >> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
> >> certificate that self-limits itself to only support Kernel Module
> >> signing.
> >> Signatures made by such certificate, are not trusted by shim for the
> >> purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
> >> linux).
> >> I also responded this on stackoverflow.
> >>
> >> The automatically generated MOK key is only usable to sign kernel
> >> modules, i.e. self-built DKMS modules.
> >>
> >> --
> >> okurrr,
> >>
> >> Dimitri
> >>
> >> On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com>
> wrote:
> >> >
> >> > Hi everyone - I'm new here!
> >> >
> >> > Sorry for going with my problem directly to the grub-devel maling
> list, but I'm pretty sure my problem is GRUB related. Still, I've spent
> some hours trying to find a solution on the Internet and I failed :( So,
> here it comes - if anyone has time to explain my problem to a layman, it
> would be awesome. Even better, if you can maybe answer here on
> stackoverflow, where it can be easier to find, I believe (
> https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature
> ).
> >> >
> >> > I'm running ubuntu with Secure Boot on. Everything works fine when I
> use a kernel that comes packaged from cannonical. Still, I have issues
> running a self-signed kernel (this is actually an externally built kernel,
> that I have verified and want to use for my own machine). I'm pretty sure
> my signature with MOK key is OK (verification below), but still when I try
> to boot the kernel from grub, after selecting the correct entry, I get an
> error that reads "Loading ... error: bad shim signature." I'm wrapping my
> head around it and can't find a solution. Why, even though both kernels are
> signed with MOK keys, one of them works and the other doesn't?
> >> >
> >> > Here's info about kernel signatures:
> >> >
> >> > root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert
> /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
> >> > Image was already signed; adding additional signature
> >> >
> >> > root@T495:~# sbverify --list /boot/vmlinuz
> >> > signature 1
> >> > image signature issuers:
> >> >  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot
> Signing/emailAddress=example@example.com
> >> > image signature certificates:
> >> >  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=example@example.com
> >> >    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
> Signing/emailAddress=example@example.com
> >> > signature 2
> >> > image signature issuers:
> >> >  - /CN=ubuntu Secure Boot Module Signature key
> >> > image signature certificates:
> >> >  - subject: /CN=ubuntu Secure Boot Module Signature key
> >> >    issuer:  /CN=ubuntu Secure Boot Module Signature key
> >> >
> >> >
> >> > And here about MOK keys:
> >> >
> >> > root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem
> -fingerprint -noout
> >> > SHA1
> Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
> >> > root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
> >> > SHA1 Fingerprint:
> 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
> >> >
> >> > If there are any docs that help understand that, I'm happy to be
> redirected there :)
> >> >
> >> > piontec
> >> > _______________________________________________
> >> > Grub-devel mailing list
> >> > Grub-devel@gnu.org
> >> > https://lists.gnu.org/mailman/listinfo/grub-devel
> >>
> >> _______________________________________________
> >> Grub-devel mailing list
> >> Grub-devel@gnu.org
> >> https://lists.gnu.org/mailman/listinfo/grub-devel
> >
> > _______________________________________________
> > Grub-devel mailing list
> > Grub-devel@gnu.org
> > https://lists.gnu.org/mailman/listinfo/grub-devel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 10101 bytes --]

Re: Can't find a solution to a failed secure boot kernel loading

[Łukasz Piątkowski]

[-- Attachment #1: Type: text/plain, Size: 7823 bytes --]

That was it, I created a new without that EKU and everything works! Thank
you very much, this was not easy to find, unfortunately :( Esp. when some
official pages like here
https://ubuntu.com/blog/how-to-sign-things-for-secure-boot still list it as
a needed EKU.

On Tue, May 10, 2022 at 4:59 PM Łukasz Piątkowski <piontec@gmail.com> wrote:

> Huh, I've never seen that before... thanks, I'm gonna give it a try and
> report back!
>
> On Tue, May 10, 2022 at 4:44 PM Dimitri John Ledkov <
> dimitri.ledkov@canonical.com> wrote:
>
>> On Tue, 10 May 2022 at 15:07, Łukasz Piątkowski <piontec@gmail.com>
>> wrote:
>> >
>> > What I'm trying to do is to sign a mainline kernel built by ubuntu (
>> https://kernel.ubuntu.com/~kernel-ppa/mainline/) with my private key,
>> that is already enrolled to MOK, and boot it with Secure Boot.
>> >
>> > > the MOK key as generated by Ubuntu/Debian tooling, creates a signing
>> certificate that self-limits itself to only support Kernel Module signing.
>> >
>> > OK, that explains why the key in `/var/lib/shim-signed/mok` doesn't
>> work. Still, I have created my own key as well (listed below for
>> inspection, it has code signing extension), enrolled that key in MOK and
>> signed the ubuntu mainline kernel (the kernel I'm trying to boot) with it.
>> The result is exactly the same. I was using exactly the same procedure a
>> few ubuntu editions back and it was definitely working. From what I learned
>> so far, this might be related to the BootHole bug (
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10713) that was
>> fixed some time ago.
>> >
>> > My generated key is:
>> >
>> > root@T495:~/mok# openssl x509 -in MOK.pem -text -noout
>> > Certificate:
>> >     Data:
>> >         Version: 3 (0x2)
>> >         Serial Number:
>> >             42:61:86:b2:29:3d:ca:eb:98:87:ae:3d:74:95:c7:f2:63:8f:8a:3b
>> >         Signature Algorithm: sha256WithRSAEncryption
>> >         Issuer: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure
>> Boot Signing, emailAddress = example@example.com
>> >         Validity
>> >             Not Before: Feb 18 19:28:16 2020 GMT
>> >             Not After : Jan 25 19:28:16 2120 GMT
>> >         Subject: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure
>> Boot Signing, emailAddress = example@example.com
>> >         Subject Public Key Info:
>> >             Public Key Algorithm: rsaEncryption
>> >                 Public-Key: (2048 bit)
>> >                 Modulus: [cut]
>> >                 Exponent: 65537 (0x10001)
>> >         X509v3 extensions:
>> >             X509v3 Subject Key Identifier:
>> >
>>  EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>> >             X509v3 Authority Key Identifier:
>> >
>>  EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>> >             X509v3 Basic Constraints: critical
>> >                 CA:FALSE
>> >             X509v3 Extended Key Usage:
>> >                 Code Signing, 1.3.6.1.4.1.311.10.3.6,
>> 1.3.6.1.4.1.2312.16.1.2
>>
>> This is bad... certs that have 1.3.6.1.4.1.2312.16.1.2 cannot be used
>> to sign kernels.
>>
>> Your cert must _not_ have 1.3.6.1.4.1.2312.16.1.2 EKU set on it.
>>
>> You cannot use the same certificate to sign both kernel and modules.
>>
>> >             Netscape Comment:
>> >                 OpenSSL Generated Certificate
>> >     Signature Algorithm: sha256WithRSAEncryption
>> >     Signature Value: [cut]
>> >
>> > On Tue, May 10, 2022 at 3:26 PM Dimitri John Ledkov <
>> dimitri.ledkov@canonical.com> wrote:
>> >>
>> >> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
>> >> certificate that self-limits itself to only support Kernel Module
>> >> signing.
>> >> Signatures made by such certificate, are not trusted by shim for the
>> >> purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
>> >> linux).
>> >> I also responded this on stackoverflow.
>> >>
>> >> The automatically generated MOK key is only usable to sign kernel
>> >> modules, i.e. self-built DKMS modules.
>> >>
>> >> --
>> >> okurrr,
>> >>
>> >> Dimitri
>> >>
>> >> On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com>
>> wrote:
>> >> >
>> >> > Hi everyone - I'm new here!
>> >> >
>> >> > Sorry for going with my problem directly to the grub-devel maling
>> list, but I'm pretty sure my problem is GRUB related. Still, I've spent
>> some hours trying to find a solution on the Internet and I failed :( So,
>> here it comes - if anyone has time to explain my problem to a layman, it
>> would be awesome. Even better, if you can maybe answer here on
>> stackoverflow, where it can be easier to find, I believe (
>> https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature
>> ).
>> >> >
>> >> > I'm running ubuntu with Secure Boot on. Everything works fine when I
>> use a kernel that comes packaged from cannonical. Still, I have issues
>> running a self-signed kernel (this is actually an externally built kernel,
>> that I have verified and want to use for my own machine). I'm pretty sure
>> my signature with MOK key is OK (verification below), but still when I try
>> to boot the kernel from grub, after selecting the correct entry, I get an
>> error that reads "Loading ... error: bad shim signature." I'm wrapping my
>> head around it and can't find a solution. Why, even though both kernels are
>> signed with MOK keys, one of them works and the other doesn't?
>> >> >
>> >> > Here's info about kernel signatures:
>> >> >
>> >> > root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert
>> /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
>> >> > Image was already signed; adding additional signature
>> >> >
>> >> > root@T495:~# sbverify --list /boot/vmlinuz
>> >> > signature 1
>> >> > image signature issuers:
>> >> >  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot
>> Signing/emailAddress=example@example.com
>> >> > image signature certificates:
>> >> >  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
>> Signing/emailAddress=example@example.com
>> >> >    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot
>> Signing/emailAddress=example@example.com
>> >> > signature 2
>> >> > image signature issuers:
>> >> >  - /CN=ubuntu Secure Boot Module Signature key
>> >> > image signature certificates:
>> >> >  - subject: /CN=ubuntu Secure Boot Module Signature key
>> >> >    issuer:  /CN=ubuntu Secure Boot Module Signature key
>> >> >
>> >> >
>> >> > And here about MOK keys:
>> >> >
>> >> > root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem
>> -fingerprint -noout
>> >> > SHA1
>> Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
>> >> > root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
>> >> > SHA1 Fingerprint:
>> 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
>> >> >
>> >> > If there are any docs that help understand that, I'm happy to be
>> redirected there :)
>> >> >
>> >> > piontec
>> >> > _______________________________________________
>> >> > Grub-devel mailing list
>> >> > Grub-devel@gnu.org
>> >> > https://lists.gnu.org/mailman/listinfo/grub-devel
>> >>
>> >> _______________________________________________
>> >> Grub-devel mailing list
>> >> Grub-devel@gnu.org
>> >> https://lists.gnu.org/mailman/listinfo/grub-devel
>> >
>> > _______________________________________________
>> > Grub-devel mailing list
>> > Grub-devel@gnu.org
>> > https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>

[-- Attachment #2: Type: text/html, Size: 10813 bytes --]

Re: Can't find a solution to a failed secure boot kernel loading

[Dimitri John Ledkov]

On Wed, 11 May 2022 at 11:14, Łukasz Piątkowski <piontec@gmail.com> wrote:
>
> That was it, I created a new without that EKU and everything works! Thank you very much, this was not easy to find, unfortunately :( Esp. when some official pages like here https://ubuntu.com/blog/how-to-sign-things-for-secure-boot still list it as a needed EKU.
>

Indeed in that blog post's section "What about kernels and
bootloaders?" is out of date, not only one needs DER cert, one needs a
cert without modules EKU set =/ I opened
https://github.com/canonical-web-and-design/ubuntu.com/issues/11595 to
see if it can be corrected.

https://wiki.ubuntu.com/UEFI/SecureBoot/KeyManagement/ImageSigning

Is the authoritative documentation (and sibling pages) that we do
maintain and keep accurate. They look correct to me.

> On Tue, May 10, 2022 at 4:59 PM Łukasz Piątkowski <piontec@gmail.com> wrote:
>>
>> Huh, I've never seen that before... thanks, I'm gonna give it a try and report back!
>>
>> On Tue, May 10, 2022 at 4:44 PM Dimitri John Ledkov <dimitri.ledkov@canonical.com> wrote:
>>>
>>> On Tue, 10 May 2022 at 15:07, Łukasz Piątkowski <piontec@gmail.com> wrote:
>>> >
>>> > What I'm trying to do is to sign a mainline kernel built by ubuntu (https://kernel.ubuntu.com/~kernel-ppa/mainline/) with my private key, that is already enrolled to MOK, and boot it with Secure Boot.
>>> >
>>> > > the MOK key as generated by Ubuntu/Debian tooling, creates a signing certificate that self-limits itself to only support Kernel Module signing.
>>> >
>>> > OK, that explains why the key in `/var/lib/shim-signed/mok` doesn't work. Still, I have created my own key as well (listed below for inspection, it has code signing extension), enrolled that key in MOK and signed the ubuntu mainline kernel (the kernel I'm trying to boot) with it. The result is exactly the same. I was using exactly the same procedure a few ubuntu editions back and it was definitely working. From what I learned so far, this might be related to the BootHole bug (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10713) that was fixed some time ago.
>>> >
>>> > My generated key is:
>>> >
>>> > root@T495:~/mok# openssl x509 -in MOK.pem -text -noout
>>> > Certificate:
>>> >     Data:
>>> >         Version: 3 (0x2)
>>> >         Serial Number:
>>> >             42:61:86:b2:29:3d:ca:eb:98:87:ae:3d:74:95:c7:f2:63:8f:8a:3b
>>> >         Signature Algorithm: sha256WithRSAEncryption
>>> >         Issuer: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure Boot Signing, emailAddress = example@example.com
>>> >         Validity
>>> >             Not Before: Feb 18 19:28:16 2020 GMT
>>> >             Not After : Jan 25 19:28:16 2120 GMT
>>> >         Subject: C = PL, ST = Poznan, L = Poznan, O = none, CN = Secure Boot Signing, emailAddress = example@example.com
>>> >         Subject Public Key Info:
>>> >             Public Key Algorithm: rsaEncryption
>>> >                 Public-Key: (2048 bit)
>>> >                 Modulus: [cut]
>>> >                 Exponent: 65537 (0x10001)
>>> >         X509v3 extensions:
>>> >             X509v3 Subject Key Identifier:
>>> >                 EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>>> >             X509v3 Authority Key Identifier:
>>> >                 EC:57:4E:BD:DC:1A:CF:B4:55:16:4A:CE:CB:E4:9E:44:5C:C4:63:F6
>>> >             X509v3 Basic Constraints: critical
>>> >                 CA:FALSE
>>> >             X509v3 Extended Key Usage:
>>> >                 Code Signing, 1.3.6.1.4.1.311.10.3.6, 1.3.6.1.4.1.2312.16.1.2
>>>
>>> This is bad... certs that have 1.3.6.1.4.1.2312.16.1.2 cannot be used
>>> to sign kernels.
>>>
>>> Your cert must _not_ have 1.3.6.1.4.1.2312.16.1.2 EKU set on it.
>>>
>>> You cannot use the same certificate to sign both kernel and modules.
>>>
>>> >             Netscape Comment:
>>> >                 OpenSSL Generated Certificate
>>> >     Signature Algorithm: sha256WithRSAEncryption
>>> >     Signature Value: [cut]
>>> >
>>> > On Tue, May 10, 2022 at 3:26 PM Dimitri John Ledkov <dimitri.ledkov@canonical.com> wrote:
>>> >>
>>> >> the MOK key as generated by Ubuntu/Debian tooling, creates a signing
>>> >> certificate that self-limits itself to only support Kernel Module
>>> >> signing.
>>> >> Signatures made by such certificate, are not trusted by shim for the
>>> >> purpose of code signing of bootloaders (i.e. grub) or kernels (i.e.
>>> >> linux).
>>> >> I also responded this on stackoverflow.
>>> >>
>>> >> The automatically generated MOK key is only usable to sign kernel
>>> >> modules, i.e. self-built DKMS modules.
>>> >>
>>> >> --
>>> >> okurrr,
>>> >>
>>> >> Dimitri
>>> >>
>>> >> On Tue, 10 May 2022 at 11:33, Łukasz Piątkowski <piontec@gmail.com> wrote:
>>> >> >
>>> >> > Hi everyone - I'm new here!
>>> >> >
>>> >> > Sorry for going with my problem directly to the grub-devel maling list, but I'm pretty sure my problem is GRUB related. Still, I've spent some hours trying to find a solution on the Internet and I failed :( So, here it comes - if anyone has time to explain my problem to a layman, it would be awesome. Even better, if you can maybe answer here on stackoverflow, where it can be easier to find, I believe (https://unix.stackexchange.com/questions/701612/cant-load-self-signed-kernel-with-secure-boot-on-bad-shim-signature).
>>> >> >
>>> >> > I'm running ubuntu with Secure Boot on. Everything works fine when I use a kernel that comes packaged from cannonical. Still, I have issues running a self-signed kernel (this is actually an externally built kernel, that I have verified and want to use for my own machine). I'm pretty sure my signature with MOK key is OK (verification below), but still when I try to boot the kernel from grub, after selecting the correct entry, I get an error that reads "Loading ... error: bad shim signature." I'm wrapping my head around it and can't find a solution. Why, even though both kernels are signed with MOK keys, one of them works and the other doesn't?
>>> >> >
>>> >> > Here's info about kernel signatures:
>>> >> >
>>> >> > root@T495:~# sbsign --key /var/lib/shim-signed/mok/MOK.priv --cert /var/lib/shim-signed/mok/MOK.pem /boot/vmlinuz
>>> >> > Image was already signed; adding additional signature
>>> >> >
>>> >> > root@T495:~# sbverify --list /boot/vmlinuz
>>> >> > signature 1
>>> >> > image signature issuers:
>>> >> >  - /C=PL/ST=Poznan/L=Poznan/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>>> >> > image signature certificates:
>>> >> >  - subject: /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>>> >> >    issuer:  /C=PL/ST=yes/L=yes/O=none/CN=Secure Boot Signing/emailAddress=example@example.com
>>> >> > signature 2
>>> >> > image signature issuers:
>>> >> >  - /CN=ubuntu Secure Boot Module Signature key
>>> >> > image signature certificates:
>>> >> >  - subject: /CN=ubuntu Secure Boot Module Signature key
>>> >> >    issuer:  /CN=ubuntu Secure Boot Module Signature key
>>> >> >
>>> >> >
>>> >> > And here about MOK keys:
>>> >> >
>>> >> > root@T495:~# openssl x509 -in /var/lib/shim-signed/mok/MOK.pem -fingerprint -noout
>>> >> > SHA1 Fingerprint=81:A2:93:CB:06:6F:52:BA:D9:E2:39:68:9D:FA:E2:2B:0C:95:3C:F7
>>> >> > root@T495:~# mokutil --list-enrolled | grep "81:a2:93"
>>> >> > SHA1 Fingerprint: 81:a2:93:cb:06:6f:52:ba:d9:e2:39:68:9d:fa:e2:2b:0c:95:3c:f7
>>> >> >
>>> >> > If there are any docs that help understand that, I'm happy to be redirected there :)
>>> >> >
>>> >> > piontec
>>> >> > _______________________________________________
>>> >> > Grub-devel mailing list
>>> >> > Grub-devel@gnu.org
>>> >> > https://lists.gnu.org/mailman/listinfo/grub-devel
>>> >>
>>> >> _______________________________________________
>>> >> Grub-devel mailing list
>>> >> Grub-devel@gnu.org
>>> >> https://lists.gnu.org/mailman/listinfo/grub-devel
>>> >
>>> > _______________________________________________
>>> > Grub-devel mailing list
>>> > Grub-devel@gnu.org
>>> > https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel