[U-Boot] UBI/UBIFS complete integrity check

[U-Boot] UBI/UBIFS complete integrity check

[Liam Beguin]

Hi everyone,

I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
and would like to perform a full integrity check before booting it.
The rootfs is read-only and until now, I've been computing an md5sum on
the whole mtd device from an initramfs and comparing it to a stored
md5sum. If both md5sums don't match, I need to stop the boot process
completely.

If possible, I was hoping to drop initramfs and do the integrity check
from U-Boot. I know UBI/UBIFS does a CRC-32 of the data it writes to
flash but the intent here is to prevent booting an image where
even a _single bit_ of flash may have been corrupted.

My question is, does UBI/UBIFS have this kind of complete integrity
check built-in? If not, can I take advantage of these CRC-32, to do
something equivalent to my md5sum check from U-Boot.
Thanks,

Liam Beguin
Xiphos Systems Corp.
http://xiphos.com

[U-Boot] UBI/UBIFS complete integrity check

[Lukasz Majewski]

Hi Liam,

> Hi everyone,
> 
> I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
> and would like to perform a full integrity check before booting it.
> The rootfs is read-only and until now, I've been computing an md5sum
> on the whole mtd device from an initramfs and comparing it to a stored
> md5sum. If both md5sums don't match, I need to stop the boot process
> completely.
> 
> If possible, I was hoping to drop initramfs and do the integrity check
> from U-Boot. 

U-boot has support for crc32 and sha1 (256). It should be possible to
do the integrity checking in it.

If you have more SDRAM than SPI-NOR, then you can calculate sha1/crc32
of the whole memory.

> I know UBI/UBIFS does a CRC-32 of the data it writes to
> flash but the intent here is to prevent booting an image where
> even a _single bit_ of flash may have been corrupted.

Ok. I see.

> 
> My question is, does UBI/UBIFS have this kind of complete integrity
> check built-in? 

As fair as I'm aware - not. The only recent improvement was the
"encryption/decryption" support.

> If not, can I take advantage of these CRC-32, 

It may be hard to access UBI metadata (from PEB/LEB).

> to do
> something equivalent to my md5sum check from U-Boot.

It may be possible to read the whole SPI-NOR Memory content to RAM,
calculate crc32/sha1 and compare with some stored value (e.g. in u-boot
envs). This all should be done with u-boot prompt.

> Thanks,
> 
> Liam Beguin
> Xiphos Systems Corp.
> http://xiphos.com
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot



Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171104/547e7711/attachment.sig>

[U-Boot] UBI/UBIFS complete integrity check

[Ladislav Michl]

On Tue, Oct 31, 2017 at 11:01:21AM -0400, Liam Beguin wrote:
> Hi everyone,
> 
> I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
> and would like to perform a full integrity check before booting it.
> The rootfs is read-only and until now, I've been computing an md5sum on
> the whole mtd device from an initramfs and comparing it to a stored
> md5sum. If both md5sums don't match, I need to stop the boot process
> completely.

Above doesn't sound right even in theory as UBI layer is free to correct
bit-flips (unlikely on SPI-NOR) and shuffle eraseblocks around even
if read only filesystem is sitting on top of it. See this faq:
http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock

So, if you are computing md5sum of underlaying mtd device you might get
different checksum even for the same UBI content.

> If possible, I was hoping to drop initramfs and do the integrity check
> from U-Boot. I know UBI/UBIFS does a CRC-32 of the data it writes to
> flash but the intent here is to prevent booting an image where
> even a _single bit_ of flash may have been corrupted.
> 
> My question is, does UBI/UBIFS have this kind of complete integrity
> check built-in? If not, can I take advantage of these CRC-32, to do
> something equivalent to my md5sum check from U-Boot.
> Thanks,

There is md5sum command, question is whenever you UBI volume fits
into RAM to do calculation at once.

> Liam Beguin
> Xiphos Systems Corp.
> http://xiphos.com
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

[U-Boot] UBI/UBIFS complete integrity check

[Liam Beguin]

Hi Lukasz,

Thanks for taking the time to answer.

On 11/04/2017 05:17 PM, Lukasz Majewski wrote:
> Hi Liam,
> 
>> Hi everyone,
>>
>> I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
>> and would like to perform a full integrity check before booting it.
>> The rootfs is read-only and until now, I've been computing an md5sum
>> on the whole mtd device from an initramfs and comparing it to a stored
>> md5sum. If both md5sums don't match, I need to stop the boot process
>> completely.
>>
>> If possible, I was hoping to drop initramfs and do the integrity check
>> from U-Boot.
> 
> U-boot has support for crc32 and sha1 (256). It should be possible to
> do the integrity checking in it.
> 
> If you have more SDRAM than SPI-NOR, then you can calculate sha1/crc32
> of the whole memory.
> 
>> I know UBI/UBIFS does a CRC-32 of the data it writes to
>> flash but the intent here is to prevent booting an image where
>> even a _single bit_ of flash may have been corrupted.
> 
> Ok. I see.
> 
>>
>> My question is, does UBI/UBIFS have this kind of complete integrity
>> check built-in?
> 
> As fair as I'm aware - not. The only recent improvement was the
> "encryption/decryption" support

I don't think I have enough time right now but would this integrity check
be an interesting feature to add?

> 
>> If not, can I take advantage of these CRC-32,
> 
> It may be hard to access UBI metadata (from PEB/LEB).
> 
>> to do
>> something equivalent to my md5sum check from U-Boot.
> 
> It may be possible to read the whole SPI-NOR Memory content to RAM,
> calculate crc32/sha1 and compare with some stored value (e.g. in u-boot
> envs). This all should be done with u-boot prompt.

This was my backup plan. I should have enough RAM to do it.

> 
>> Thanks,
>>
>> Liam Beguin
>> Xiphos Systems Corp.
>> http://xiphos.com
>> _______________________________________________
>> U-Boot mailing list
>> U-Boot at lists.denx.de
>> https://lists.denx.de/listinfo/u-boot
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
> 

Thanks,

Liam Beguin
Xiphos Systems Corp.
http://xiphos.com

[U-Boot] UBI/UBIFS complete integrity check

[Lukasz Majewski]

Hi Liam,

> Hi Lukasz,
> 
> Thanks for taking the time to answer.
> 
> On 11/04/2017 05:17 PM, Lukasz Majewski wrote:
> > Hi Liam,
> >   
> >> Hi everyone,
> >>
> >> I'm currently using a UBIFS root file system (stored on SPI-NOR
> >> flash) and would like to perform a full integrity check before
> >> booting it. The rootfs is read-only and until now, I've been
> >> computing an md5sum on the whole mtd device from an initramfs and
> >> comparing it to a stored md5sum. If both md5sums don't match, I
> >> need to stop the boot process completely.
> >>
> >> If possible, I was hoping to drop initramfs and do the integrity
> >> check from U-Boot.  
> > 
> > U-boot has support for crc32 and sha1 (256). It should be possible
> > to do the integrity checking in it.
> > 
> > If you have more SDRAM than SPI-NOR, then you can calculate
> > sha1/crc32 of the whole memory.
> >   
> >> I know UBI/UBIFS does a CRC-32 of the data it writes to
> >> flash but the intent here is to prevent booting an image where
> >> even a _single bit_ of flash may have been corrupted.  
> > 
> > Ok. I see.
> >   
> >>
> >> My question is, does UBI/UBIFS have this kind of complete integrity
> >> check built-in?  
> > 
> > As fair as I'm aware - not. The only recent improvement was the
> > "encryption/decryption" support  
> 
> I don't think I have enough time right now but would this integrity
> check be an interesting feature to add?

It depends how "secure" your project needs to be...

It is just one of the options to consider.

> 
> >   
> >> If not, can I take advantage of these CRC-32,  
> > 
> > It may be hard to access UBI metadata (from PEB/LEB).
> >   
> >> to do
> >> something equivalent to my md5sum check from U-Boot.  
> > 
> > It may be possible to read the whole SPI-NOR Memory content to RAM,
> > calculate crc32/sha1 and compare with some stored value (e.g. in
> > u-boot envs). This all should be done with u-boot prompt.  
> 
> This was my backup plan. I should have enough RAM to do it.

Ok. Good.

> 
> >   
> >> Thanks,
> >>
> >> Liam Beguin
> >> Xiphos Systems Corp.
> >> http://xiphos.com
> >> _______________________________________________
> >> U-Boot mailing list
> >> U-Boot at lists.denx.de
> >> https://lists.denx.de/listinfo/u-boot  
> > 
> > 
> > 
> > Best regards,
> > 
> > Lukasz Majewski
> > 
> > --
> > 
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> > wd at denx.de 
> 
> Thanks,
> 
> Liam Beguin
> Xiphos Systems Corp.
> http://xiphos.com
> 



Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171106/9dcffa00/attachment.sig>

[U-Boot] UBI/UBIFS complete integrity check

[Liam Beguin]

Hi,

Thanks for taking the time to answer.

On 11/05/2017 03:37 AM, Ladislav Michl wrote:
> On Tue, Oct 31, 2017 at 11:01:21AM -0400, Liam Beguin wrote:
>> Hi everyone,
>>
>> I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
>> and would like to perform a full integrity check before booting it.
>> The rootfs is read-only and until now, I've been computing an md5sum on
>> the whole mtd device from an initramfs and comparing it to a stored
>> md5sum. If both md5sums don't match, I need to stop the boot process
>> completely.
> 
> Above doesn't sound right even in theory as UBI layer is free to correct
> bit-flips (unlikely on SPI-NOR) and shuffle eraseblocks around even
> if read only filesystem is sitting on top of it. See this faq:
> http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock> 
> So, if you are computing md5sum of underlaying mtd device you might get
> different checksum even for the same UBI content.
>

I forgot to mention that the flash I use has a hardware lock which is enabled
after the filesystem is first written (the flash is locked during boot).
I'm quite confident this works as I've been using the md5sum mechanism for
some time now.

As the UBI layer is able to detect/fix bit-flips, what happens if a bit-flip is
detected and UBI cannot write to flash? does it fail to attach?
  
>> If possible, I was hoping to drop initramfs and do the integrity check
>> from U-Boot. I know UBI/UBIFS does a CRC-32 of the data it writes to
>> flash but the intent here is to prevent booting an image where
>> even a _single bit_ of flash may have been corrupted.
>>
>> My question is, does UBI/UBIFS have this kind of complete integrity
>> check built-in? If not, can I take advantage of these CRC-32, to do
>> something equivalent to my md5sum check from U-Boot.
>> Thanks,
> 
> There is md5sum command, question is whenever you UBI volume fits
> into RAM to do calculation at once.
> 
>> Liam Beguin
>> Xiphos Systems Corp.
>> http://xiphos.com
>> _______________________________________________
>> U-Boot mailing list
>> U-Boot at lists.denx.de
>> https://lists.denx.de/listinfo/u-boot

Thanks,

Liam Beguin
Xiphos Systems Corp.
http://xiphos.com

[U-Boot] UBI/UBIFS complete integrity check

[Ladislav Michl]

On Mon, Nov 06, 2017 at 12:31:32PM -0500, Liam Beguin wrote:
> Hi,
> 
> Thanks for taking the time to answer.
> 
> On 11/05/2017 03:37 AM, Ladislav Michl wrote:
> > On Tue, Oct 31, 2017 at 11:01:21AM -0400, Liam Beguin wrote:
> > > Hi everyone,
> > > 
> > > I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
> > > and would like to perform a full integrity check before booting it.
> > > The rootfs is read-only and until now, I've been computing an md5sum on
> > > the whole mtd device from an initramfs and comparing it to a stored
> > > md5sum. If both md5sums don't match, I need to stop the boot process
> > > completely.
> > 
> > Above doesn't sound right even in theory as UBI layer is free to correct
> > bit-flips (unlikely on SPI-NOR) and shuffle eraseblocks around even
> > if read only filesystem is sitting on top of it. See this faq:
> > http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock> So, if you
> > are computing md5sum of underlaying mtd device you might get
> > different checksum even for the same UBI content.
> > 
> 
> I forgot to mention that the flash I use has a hardware lock which is enabled
> after the filesystem is first written (the flash is locked during boot).
> I'm quite confident this works as I've been using the md5sum mechanism for
> some time now.

I see. Btw, have you considered using for example squashfs on top of static UBI
volume? Given the fact your fs is read only you get:
- CRC32 checksum of you volume for free
- filesystem compression
- you can easily compute mtd5sum of volume content
- check filesystem consistency using already provided tools (probably overkill).

> As the UBI layer is able to detect/fix bit-flips, what happens if a bit-flip is
> detected and UBI cannot write to flash? does it fail to attach?

No, it just logs it. It fails once there is enough bitflips that ECC is unable
to correct them. But again, it is not case of NOR, I mentioned it just for
completeness.

> > > If possible, I was hoping to drop initramfs and do the integrity check
> > > from U-Boot. I know UBI/UBIFS does a CRC-32 of the data it writes to
> > > flash but the intent here is to prevent booting an image where
> > > even a _single bit_ of flash may have been corrupted.

Above mentioned solution should fail instantly in case of CRC error. Not sure if
that's what you want as kernel already took over, so either watchdog will kick in
or machine will reboot, if configured to do so after panics.

> > > My question is, does UBI/UBIFS have this kind of complete integrity
> > > check built-in? If not, can I take advantage of these CRC-32, to do
> > > something equivalent to my md5sum check from U-Boot.
> > > Thanks,
> > 
> > There is md5sum command, question is whenever you UBI volume fits
> > into RAM to do calculation at once.
> > 
> > > Liam Beguin
> > > Xiphos Systems Corp.
> > > http://xiphos.com
> > > _______________________________________________
> > > U-Boot mailing list
> > > U-Boot at lists.denx.de
> > > https://lists.denx.de/listinfo/u-boot
> 
> Thanks,
> 
> Liam Beguin
> Xiphos Systems Corp.
> http://xiphos.com

[U-Boot] UBI/UBIFS complete integrity check

[Liam Beguin]

On 11/06/2017 12:57 PM, Ladislav Michl wrote:
> On Mon, Nov 06, 2017 at 12:31:32PM -0500, Liam Beguin wrote:
>> Hi,
>>
>> Thanks for taking the time to answer.
>>
>> On 11/05/2017 03:37 AM, Ladislav Michl wrote:
>>> On Tue, Oct 31, 2017 at 11:01:21AM -0400, Liam Beguin wrote:
>>>> Hi everyone,
>>>>
>>>> I'm currently using a UBIFS root file system (stored on SPI-NOR flash)
>>>> and would like to perform a full integrity check before booting it.
>>>> The rootfs is read-only and until now, I've been computing an md5sum on
>>>> the whole mtd device from an initramfs and comparing it to a stored
>>>> md5sum. If both md5sums don't match, I need to stop the boot process
>>>> completely.
>>>
>>> Above doesn't sound right even in theory as UBI layer is free to correct
>>> bit-flips (unlikely on SPI-NOR) and shuffle eraseblocks around even
>>> if read only filesystem is sitting on top of it. See this faq:
>>> http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock> So, if you
>>> are computing md5sum of underlaying mtd device you might get
>>> different checksum even for the same UBI content.
>>>
>>
>> I forgot to mention that the flash I use has a hardware lock which is enabled
>> after the filesystem is first written (the flash is locked during boot).
>> I'm quite confident this works as I've been using the md5sum mechanism for
>> some time now.
> 
> I see. Btw, have you considered using for example squashfs on top of static UBI
> volume? Given the fact your fs is read only you get:
> - CRC32 checksum of you volume for free
> - filesystem compression
> - you can easily compute mtd5sum of volume content
> - check filesystem consistency using already provided tools (probably overkill).
> 

I have considered squashfs but on rare occasions, I need to be able to write to the
rootfs. This involves disabling the hardware lock, remounting RW and updating the
reference md5sum after writing.

>> As the UBI layer is able to detect/fix bit-flips, what happens if a bit-flip is
>> detected and UBI cannot write to flash? does it fail to attach?
> 
> No, it just logs it. It fails once there is enough bitflips that ECC is unable
> to correct them. But again, it is not case of NOR, I mentioned it just for
> completeness.

I understand that bit-flips are uncommon on NOR flash but in our case, they may
be caused by the environment not the flash technology.

> 
>>>> If possible, I was hoping to drop initramfs and do the integrity check
>>>> from U-Boot. I know UBI/UBIFS does a CRC-32 of the data it writes to
>>>> flash but the intent here is to prevent booting an image where
>>>> even a _single bit_ of flash may have been corrupted.
> 
> Above mentioned solution should fail instantly in case of CRC error. Not sure if
> that's what you want as kernel already took over, so either watchdog will kick in
> or machine will reboot, if configured to do so after panics.
> 

Ideally, I'd like the attach to fail in U-Boot, before loading the kernel.

Something remains obscure to me... When UBI scans the device and potentially
warns of a bit-flip, has the _whole_ data been checked or was it only the
information contained in the UBI headers?

>>>> My question is, does UBI/UBIFS have this kind of complete integrity
>>>> check built-in? If not, can I take advantage of these CRC-32, to do
>>>> something equivalent to my md5sum check from U-Boot.
>>>> Thanks,
>>>
>>> There is md5sum command, question is whenever you UBI volume fits
>>> into RAM to do calculation at once.
>>>
>>>> Liam Beguin
>>>> Xiphos Systems Corp.
>>>> http://xiphos.com
>>>> _______________________________________________
>>>> U-Boot mailing list
>>>> U-Boot at lists.denx.de
>>>> https://lists.denx.de/listinfo/u-boot
>>
>> Thanks,
>>
>> Liam Beguin
>> Xiphos Systems Corp.
>> http://xiphos.com