Verifier running out of memory on ieee1275/powerpc64

Verifier running out of memory on ieee1275/powerpc64

[Stefan Berger]

Hello!

  I trying to add (v)TPM support for the ieee1275/powerpc64 platform to 
grub. The issue I have been running into is that the verifier runs out 
of memory. At that point it has loaded the (~ 32MB) Linux kernel and now 
the verifier is invoked to load the file. Unfortunately it cannot load 
the file since it doesn't have enough memory to grub_malloc. I have 
played with increasing heap size(es) but it still doesn't work. The 
kernel and initramfs files on ppc64 can be rather big, thus we do not a 
lot of memory. The rescue initramfs here is for example 78MB, a regular 
initramfs from Fedora 31 is ~34MB. The kernel sizes on my system are 
32MB, though a colleague was using an unstripped kernel of 127MB, so 
lots of (unfragmented) memory needs to be available to run verifiers.


This is the code that's trying to allocate the memory to load the file 
into: 
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/commands/verifiers.c#n128

I had increased the GRUB_IEEE_STATIC_HEAP_LEN to 0x4000000, but it 
didn't resolve the issue: 
http://git.savannah.gnu.org/cgit/grub.git/tree/include/grub/ieee1275/ieee1275.h#n93

I also tried to increase the HEAP_MAX_SIZE to 128 * 1024 * 1024 along 
with HEAP_MAX_ADDR, but that didn't help, either: 
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/kern/ieee1275/init.c


Does anybody have a suggestion?


    Regards,

     Stefan

Re: Verifier running out of memory on ieee1275/powerpc64

[Simon Hardy]

The 2020/03/17 13:15, Stefan Berger wrote:
>  I trying to add (v)TPM support for the ieee1275/powerpc64 platform to grub.
> The issue I have been running into is that the verifier runs out of memory.
> At that point it has loaded the (~ 32MB) Linux kernel and now the verifier
> is invoked to load the file. Unfortunately it cannot load the file since it
> doesn't have enough memory to grub_malloc. I have played with increasing
> heap size(es) but it still doesn't work. The kernel and initramfs files on
> ppc64 can be rather big, thus we do not a lot of memory. The rescue
> initramfs here is for example 78MB, a regular initramfs from Fedora 31 is
> ~34MB. The kernel sizes on my system are 32MB, though a colleague was using
> an unstripped kernel of 127MB, so lots of (unfragmented) memory needs to be
> available to run verifiers.

The verifiers framework has a flag, GRUB_VERIFY_FLAGS_SINGLE_CHUNK, that is
used by the platform-independent TPM module. This could be deferred to the
platform-specific TPM file (see point 3 below). With this flag unset for your
platform, you could verify the files in small chunks. This requires three
further elements: 

1. You will need to implement the chunk-by-chunk behaviour in
verifiers.c, it doesn't exist yet.

2. You will need to add functionality to calculate a hash from chunks, or
require that the crypto module is built into the core.

3. The firmware interface needs to support HashLogExtend with a user supplied
hash instead of a memory buffer. For example the PC Conventional BIOS API has
this, but the UEFI API does not.

Re: Verifier running out of memory on ieee1275/powerpc64

[Stefan Berger]

On 3/18/20 11:59 AM, Simon Hardy wrote:
> The 2020/03/17 13:15, Stefan Berger wrote:
>>   I trying to add (v)TPM support for the ieee1275/powerpc64 platform to grub.
>> The issue I have been running into is that the verifier runs out of memory.
>> At that point it has loaded the (~ 32MB) Linux kernel and now the verifier
>> is invoked to load the file. Unfortunately it cannot load the file since it
>> doesn't have enough memory to grub_malloc. I have played with increasing
>> heap size(es) but it still doesn't work. The kernel and initramfs files on
>> ppc64 can be rather big, thus we do not a lot of memory. The rescue
>> initramfs here is for example 78MB, a regular initramfs from Fedora 31 is
>> ~34MB. The kernel sizes on my system are 32MB, though a colleague was using
>> an unstripped kernel of 127MB, so lots of (unfragmented) memory needs to be
>> available to run verifiers.
> The verifiers framework has a flag, GRUB_VERIFY_FLAGS_SINGLE_CHUNK, that is
> used by the platform-independent TPM module. This could be deferred to the
> platform-specific TPM file (see point 3 below). With this flag unset for your
> platform, you could verify the files in small chunks. This requires three
> further elements:
>
> 1. You will need to implement the chunk-by-chunk behaviour in
> verifiers.c, it doesn't exist yet.
>
> 2. You will need to add functionality to calculate a hash from chunks, or
> require that the crypto module is built into the core.
>
> 3. The firmware interface needs to support HashLogExtend with a user supplied
> hash instead of a memory buffer. For example the PC Conventional BIOS API has
> this, but the UEFI API does not.

Simon, thanks a lot for your reply.

TPM 2's logging behavior is different than that of a TPM 1.2 and it's 
not clear whether hashing in grub will produce the right hash or hashes 
(for different PCR banks) as needed by the firmware. I would rather 
leave the hashing entirely up to the firmware because it knows which PCR 
banks are activate and what hashes it wants to use for logging and PCR 
extending.

I was wondering whether it would not be possible to load the raw file 
into memory, pass it to the firmware for hashing (and logging) via the 
verifier, and if we do not trust that the firmware treated the file data 
as a read-only array, load the file again into the same array right 
after. This way we wouldn't need more memory. [*] However, I am not sure 
how it fits into the architecture with the verifiers or whether the TPM 
verifier would have to take on a special role (possibly with a flag) then.

You didn't pick up on the idea of a bigger heap. Is there a problem with 
the heap size somehow? My machine has GBs of memory, so it really 
shouldn't be a problem to get memory.


[*] Obviously we can load the files. We can also load them while the TPM 
verifier is active but only when preventing the loading of those large 
files (actually allowing the grub_malloc() on the large size to occurr 
seems to cause issues booting). The memory allocations that are 
occurring due to the TPM verifier do not seem to cause memory 
fragmentation  that would prevent the loading of those large files. So, 
makes me think, re-load the files into the same memory.


    Stefan



>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: Verifier running out of memory on ieee1275/powerpc64

[Stefan Berger]

On 3/18/20 3:32 PM, Stefan Berger wrote:
> On 3/18/20 11:59 AM, Simon Hardy wrote:
>> The 2020/03/17 13:15, Stefan Berger wrote:
>>>   I trying to add (v)TPM support for the ieee1275/powerpc64 platform 
>>> to grub.
>>> The issue I have been running into is that the verifier runs out of 
>>> memory.
>>> At that point it has loaded the (~ 32MB) Linux kernel and now the 
>>> verifier
>>> is invoked to load the file. Unfortunately it cannot load the file 
>>> since it
>>> doesn't have enough memory to grub_malloc. I have played with 
>>> increasing
>>> heap size(es) but it still doesn't work. The kernel and initramfs 
>>> files on
>>> ppc64 can be rather big, thus we do not a lot of memory. The rescue
>>> initramfs here is for example 78MB, a regular initramfs from Fedora 
>>> 31 is
>>> ~34MB. The kernel sizes on my system are 32MB, though a colleague 
>>> was using
>>> an unstripped kernel of 127MB, so lots of (unfragmented) memory 
>>> needs to be
>>> available to run verifiers.
>> The verifiers framework has a flag, GRUB_VERIFY_FLAGS_SINGLE_CHUNK, 
>> that is
>> used by the platform-independent TPM module. This could be deferred 
>> to the
>> platform-specific TPM file (see point 3 below). With this flag unset 
>> for your
>> platform, you could verify the files in small chunks. This requires 
>> three
>> further elements:
>>
>> 1. You will need to implement the chunk-by-chunk behaviour in
>> verifiers.c, it doesn't exist yet.
>>
>> 2. You will need to add functionality to calculate a hash from 
>> chunks, or
>> require that the crypto module is built into the core.
>>
>> 3. The firmware interface needs to support HashLogExtend with a user 
>> supplied
>> hash instead of a memory buffer. For example the PC Conventional BIOS 
>> API has
>> this, but the UEFI API does not.
>
> Simon, thanks a lot for your reply.
>
> TPM 2's logging behavior is different than that of a TPM 1.2 and it's 
> not clear whether hashing in grub will produce the right hash or 
> hashes (for different PCR banks) as needed by the firmware. I would 
> rather leave the hashing entirely up to the firmware because it knows 
> which PCR banks are activate and what hashes it wants to use for 
> logging and PCR extending.
>
> I was wondering whether it would not be possible to load the raw file 
> into memory, pass it to the firmware for hashing (and logging) via the 
> verifier, and if we do not trust that the firmware treated the file 
> data as a read-only array, load the file again into the same array 
> right after. This way we wouldn't need more memory. [*] However, I am 
> not sure how it fits into the architecture with the verifiers or 
> whether the TPM verifier would have to take on a special role 
> (possibly with a flag) then.
>
> You didn't pick up on the idea of a bigger heap. Is there a problem 
> with the heap size somehow? My machine has GBs of memory, so it really 
> shouldn't be a problem to get memory.

I think that's the problem to solve, at least for this platform, since 
none of the verifiers will work due to the memory exhaustion issue.


    Stefan

Re: Verifier running out of memory on ieee1275/powerpc64

[Simon Hardy]

> > I was wondering whether it would not be possible to load the raw file
> > into memory, pass it to the firmware for hashing (and logging) via the
> > verifier, and if we do not trust that the firmware treated the file data
> > as a read-only array, load the file again into the same array right
> > after. This way we wouldn't need more memory. [*] However, I am not sure
> > how it fits into the architecture with the verifiers or whether the TPM
> > verifier would have to take on a special role (possibly with a flag)
> > then.
> > 
> > You didn't pick up on the idea of a bigger heap. Is there a problem with
> > the heap size somehow? My machine has GBs of memory, so it really
> > shouldn't be a problem to get memory.
> 
> I think that's the problem to solve, at least for this platform, since none
> of the verifiers will work due to the memory exhaustion issue.

Increasing the heap size may be the easiest way for you to solve this
problem. I am not so familiar with the issues around doing this, but I
have seen recent discussion on this list.

The reason there is extra memory usage is that the verifier framework is
built on top of the file filter framework. The process is:
1. The file is opened using grub_file_open.
2. The verifier framework opens the underlying file and reads all of it
into a new heap-allocated memory buffer.
3. The verifier framework provides the buffer to the verifiers for
verification.
4. The verifier framework returns a file handle.
5. Read operations on this file handle are served by copying from the
verification buffer.
6. The file is closed. The verification framework frees its buffer.

There are good reasons for doing this. It is more secure to ensure that
verification is applied to the same data that is actually used. (An
attacker could give you different data on a second read from the disk.)
Some verifiers, like the TPM, prefer to receive the data in a single
chunk. Existing code depends on the open/read/seek/close file access
model.

The problem you have run into is that where you have a use case that
involves loading a large file into memory then you need enough free
memory to hold two copies of that file.

Ideally there would be an alternative grub_file_read_all function that
loads the contents of a named file to a user-provided memory buffer.
This doesn't look like it would be simple or easy, for example I don't
know how this would work for compressed files.

Re: Verifier running out of memory on ieee1275/powerpc64

[Stefan Berger]

On 3/18/20 6:17 PM, Simon Hardy wrote:
>>> I was wondering whether it would not be possible to load the raw file
>>> into memory, pass it to the firmware for hashing (and logging) via the
>>> verifier, and if we do not trust that the firmware treated the file data
>>> as a read-only array, load the file again into the same array right
>>> after. This way we wouldn't need more memory. [*] However, I am not sure
>>> how it fits into the architecture with the verifiers or whether the TPM
>>> verifier would have to take on a special role (possibly with a flag)
>>> then.
>>>
>>> You didn't pick up on the idea of a bigger heap. Is there a problem with
>>> the heap size somehow? My machine has GBs of memory, so it really
>>> shouldn't be a problem to get memory.
>> I think that's the problem to solve, at least for this platform, since none
>> of the verifiers will work due to the memory exhaustion issue.
> Increasing the heap size may be the easiest way for you to solve this
> problem. I am not so familiar with the issues around doing this, but I
> have seen recent discussion on this list.

I found the lsmmap command and it shows me some impressive memory ranges 
to be known:

base_addr = 0x4000, length = 0x17c000, available RAM

base_addr = 0x1fffff, length = 0x1, available RAM

base_addr = 0x21ffff, length = 0x1, available RAM

base_addr = 0x2000000, length = 0x7bbf0000, available RAM

base_addr = 0x7feffffff, length = 0x1, available RAM

base_addr = 0x800903dc, length = 0x7ff6fc24, available RAM

There's more than needed memory here. It doesn't seem to be available 
for grub_malloc, though.

>
> The reason there is extra memory usage is that the verifier framework is
> built on top of the file filter framework. The process is:
> 1. The file is opened using grub_file_open.
> 2. The verifier framework opens the underlying file and reads all of it
> into a new heap-allocated memory buffer.
> 3. The verifier framework provides the buffer to the verifiers for
> verification.
> 4. The verifier framework returns a file handle.
> 5. Read operations on this file handle are served by copying from the
> verification buffer.
Yes, saw this, verify_fs.
> 6. The file is closed. The verification framework frees its buffer.
>
> There are good reasons for doing this. It is more secure to ensure that
> verification is applied to the same data that is actually used. (An
> attacker could give you different data on a second read from the disk.)
Also true.
> Some verifiers, like the TPM, prefer to receive the data in a single
> chunk. Existing code depends on the open/read/seek/close file access
> model.
>
> The problem you have run into is that where you have a use case that
> involves loading a large file into memory then you need enough free
> memory to hold two copies of that file.
>
> Ideally there would be an alternative grub_file_read_all function that
> loads the contents of a named file to a user-provided memory buffer.
> This doesn't look like it would be simple or easy, for example I don't
> know how this would work for compressed files.


Stefan


>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: Verifier running out of memory on ieee1275/powerpc64

[Eric Snowberg]

> On Mar 19, 2020, at 10:37 AM, Stefan Berger <stefanb@linux.ibm.com> wrote:
> 
> On 3/18/20 6:17 PM, Simon Hardy wrote:
>>>> I was wondering whether it would not be possible to load the raw file
>>>> into memory, pass it to the firmware for hashing (and logging) via the
>>>> verifier, and if we do not trust that the firmware treated the file data
>>>> as a read-only array, load the file again into the same array right
>>>> after. This way we wouldn't need more memory. [*] However, I am not sure
>>>> how it fits into the architecture with the verifiers or whether the TPM
>>>> verifier would have to take on a special role (possibly with a flag)
>>>> then.
>>>> 
>>>> You didn't pick up on the idea of a bigger heap. Is there a problem with
>>>> the heap size somehow? My machine has GBs of memory, so it really
>>>> shouldn't be a problem to get memory.
>>> I think that's the problem to solve, at least for this platform, since none
>>> of the verifiers will work due to the memory exhaustion issue.
>> Increasing the heap size may be the easiest way for you to solve this
>> problem. I am not so familiar with the issues around doing this, but I
>> have seen recent discussion on this list.
> 
> I found the lsmmap command and it shows me some impressive memory ranges to be known:
> 
> base_addr = 0x4000, length = 0x17c000, available RAM
> 
> base_addr = 0x1fffff, length = 0x1, available RAM
> 
> base_addr = 0x21ffff, length = 0x1, available RAM
> 
> base_addr = 0x2000000, length = 0x7bbf0000, available RAM
> 
> base_addr = 0x7feffffff, length = 0x1, available RAM
> 
> base_addr = 0x800903dc, length = 0x7ff6fc24, available RAM
> 
> There's more than needed memory here. It doesn't seem to be available for grub_malloc, though.

On SPARC, I thought the code was never written to claim memory in different regions.
Does that same problem exist on PPC?

I ran into a lot of memory corruption problems with ofdisk.  I don’t know if this
will solve your problem, but you could try this patch:

https://lists.gnu.org/archive/html/grub-devel/2016-06/msg00035.html

I couldn’t get any new ieee1275 code to work without it.