wic creates ext4 images that read really slow in u-boot

wic creates ext4 images that read really slow in u-boot

[Mike Looijmans]

Took me a while to figure out why my board took 90 seconds to boot suddenly.

The issue turned out to be the ext4 partition created by wic.

On a "wic" created ext4 disk, this is what u-boot does:

ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
19311092 bytes read in 85529 ms (219.7 KiB/s)


Now if I boot the board rename and copy that file onto itself, then it's 
suddenly normal again when I reboot the board:

ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
19311092 bytes read in 1426 ms (12.9 MiB/s)


I checked with other files, and it turns out that ANY big file written by wic 
will read very slow (at 200k/s) but if I make a copy from a running system, it 
suddenly reads fast.

I'm not knowledgeable on ext4, so any ideas what's being passed onto the image 
creation tool that causes this?

Re: wic creates ext4 images that read really slow in u-boot

[Leon Woestenberg]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

Hello Mike,

sounds familiar.

On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl>
wrote:

> Took me a while to figure out why my board took 90 seconds to boot
> suddenly.
>
> The issue turned out to be the ext4 partition created by wic.


I suspect it's not WIC's fault.


> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>
> Now if I boot the board rename and copy that file onto itself, then it's
> suddenly normal again when I reboot the board:
>
> ZynqMP> load mmc 0:2 0x10000
> I'm not knowledgeable on ext4, so any ideas what's being passed onto the
> image
> creation tool that causes this?


I suspect your version of U-Boot does not handle files spread across
multiple filesystems (allocation) extends efficiently.

Copying the file makes the copy being layed out in one extend probably.

I am aware of two fixes for U-Boot. I will look them up, and reply again to
this thread.

Regards, Leon
-- 
Leon Woestenberg
leon@sidebranch.com
T: +31 40 711 42 76
M: +31 6 472 30 372

Sidebranch
Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com

[-- Attachment #2: Type: text/html, Size: 3164 bytes --]

Re: wic creates ext4 images that read really slow in u-boot

[Andre McCurdy]

[-- Attachment #1: Type: text/plain, Size: 1637 bytes --]

On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com>
wrote:

> Hello Mike,
>
> sounds familiar.
>
> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl>
> wrote:
>
>> Took me a while to figure out why my board took 90 seconds to boot
>> suddenly.
>>
>> The issue turned out to be the ext4 partition created by wic.
>
>
> I suspect it's not WIC's fault.
>
>
>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>
>> Now if I boot the board rename and copy that file onto itself, then it's
>> suddenly normal again when I reboot the board:
>>
>> ZynqMP> load mmc 0:2 0x10000
>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the
>> image
>> creation tool that causes this?
>
>
> I suspect your version of U-Boot does not handle files spread across
> multiple filesystems (allocation) extends efficiently.
>
> Copying the file makes the copy being layed out in one extend probably.
>

If WIC is generating filesystem images from scratch there's no excuse for
files to be unnecessarily fragmented.

Even if some of all of the boot time can be recovered by a patch to u-boot
that won't help systems which have already been deployed and don't have a
way to update the bootloader.

I am aware of two fixes for U-Boot. I will look them up, and reply again to
> this thread.
>
> Regards, Leon
> --
> Leon Woestenberg
> leon@sidebranch.com
> T: +31 40 711 42 76
> M: +31 6 472 30 372
>
> Sidebranch
> Embedded Systems
> Eindhoven, The Netherlands
> http://www.sidebranch.com
>

[-- Attachment #2: Type: text/html, Size: 3944 bytes --]

Re: wic creates ext4 images that read really slow in u-boot

[Leon Woestenberg]

Hello all,

On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
>>
>> Hello Mike,
>>
>> sounds familiar.
>>
>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>>
>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
>>> The issue turned out to be the ext4 partition created by wic.
>>
>> I suspect it's not WIC's fault.
>>>
>>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>>
>>> Now if I boot the board rename and copy that file onto itself, then it's
>>> suddenly normal again when I reboot the board:
>>>
>>> ZynqMP> load mmc 0:2 0x10000
>>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
>>> creation tool that causes this?
>>
>> I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
>>
>> Copying the file makes the copy being layed out in one extend probably.
>
>
> If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
>
> Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
>
I am not sure if "fragmented" is the right term in terms of filesystem
details. Filesystem allocation extents (not "extends" as I mistyped
earlier) are different from heavy file fragmentation. However, I agree
things can be made more optimal.

So, with adding the above, there are *two* issues at play here:
1) Most/older versions of U-Boot not able to efficiently load files
from ext4 filesystems, that cross multiple extents. I am aware of two
fixes, see below.
2) WIC uses mkext4fs in such a way that files can cross multiple
(allocation) extents. This is sub-optimal and know to cause issues
with many U-Boot versions (deployed on existing systems in the field).
The problems shows "randomly" with large files being deployed to the
ext4 filesystem. We (OE/Yocto) may want to fix this.

>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.

Google for these two commits. I cherry-picked the first one for my
project and can confirm it brings back the desired performance. (Still
assuming this is the identical cause as for TO/Mike.)

commit 855b8e4f9f255415a7753819e392e4b5d984f35c
Author: Matt Madison <matt@madison.systems>
Date:   Sat Aug 19 08:46:46 2017 -0700

    ext4: cache extent blocks during file reads

    A simpler and less-efficient approach to solving
    the problem with extent index blocks than what
    was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
    but one that does not break write operations.

    Signed-off-by: Matt Madison <matt@madison.systems>

Regards,

Leon.

p.s. excuse the earlier HTML mail with signature

Re: wic creates ext4 images that read really slow in u-boot

[Jack Mitchell]

On 19/02/2019 20:45, Leon Woestenberg wrote:
> Hello all,
> 
> On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
>>>
>>> Hello Mike,
>>>
>>> sounds familiar.
>>>
>>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>>>
>>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
>>>> The issue turned out to be the ext4 partition created by wic.
>>>
>>> I suspect it's not WIC's fault.
>>>>
>>>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>>>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>>>
>>>> Now if I boot the board rename and copy that file onto itself, then it's
>>>> suddenly normal again when I reboot the board:
>>>>
>>>> ZynqMP> load mmc 0:2 0x10000
>>>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
>>>> creation tool that causes this?
>>>
>>> I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
>>>
>>> Copying the file makes the copy being layed out in one extend probably.
>>
>>
>> If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
>>
>> Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
>>
> I am not sure if "fragmented" is the right term in terms of filesystem
> details. Filesystem allocation extents (not "extends" as I mistyped
> earlier) are different from heavy file fragmentation. However, I agree
> things can be made more optimal.
> 
> So, with adding the above, there are *two* issues at play here:
> 1) Most/older versions of U-Boot not able to efficiently load files
> from ext4 filesystems, that cross multiple extents. I am aware of two
> fixes, see below.
> 2) WIC uses mkext4fs in such a way that files can cross multiple
> (allocation) extents. This is sub-optimal and know to cause issues
> with many U-Boot versions (deployed on existing systems in the field).
> The problems shows "randomly" with large files being deployed to the
> ext4 filesystem. We (OE/Yocto) may want to fix this.
> 
>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
> 
> Google for these two commits. I cherry-picked the first one for my
> project and can confirm it brings back the desired performance. (Still
> assuming this is the identical cause as for TO/Mike.)
> 
> commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> Author: Matt Madison <matt@madison.systems>
> Date:   Sat Aug 19 08:46:46 2017 -0700
> 
>     ext4: cache extent blocks during file reads
> 
>     A simpler and less-efficient approach to solving
>     the problem with extent index blocks than what
>     was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
>     but one that does not break write operations.
> 
>     Signed-off-by: Matt Madison <matt@madison.systems>
> 
> Regards,
> 
> Leon.
> 
> p.s. excuse the earlier HTML mail with signature
> 

I can concur that I have also used this patch to fix speed issues with
ext4 reads. The problem becomes particularly prominent when you have
large files on a small partition which get over-written. I'm sure I have
seen a patch go into uboot mainline recently which improved performance
but I can't find it or it never actually made it into the tree.

Re: wic creates ext4 images that read really slow in u-boot

[Mike Looijmans]

On 19-02-19 21:45, Leon Woestenberg wrote:
> Hello all,
> 
> On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
>>>
>>> Hello Mike,
>>>
>>> sounds familiar.
>>>
>>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>>>
>>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
>>>> The issue turned out to be the ext4 partition created by wic.
>>>
>>> I suspect it's not WIC's fault.
>>>>
>>>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>>>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>>>
>>>> Now if I boot the board rename and copy that file onto itself, then it's
>>>> suddenly normal again when I reboot the board:
>>>>
>>>> ZynqMP> load mmc 0:2 0x10000
>>>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
>>>> creation tool that causes this?
>>>
>>> I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
>>>
>>> Copying the file makes the copy being layed out in one extend probably.
>>
>>
>> If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
>>
>> Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
>>
> I am not sure if "fragmented" is the right term in terms of filesystem
> details. Filesystem allocation extents (not "extends" as I mistyped
> earlier) are different from heavy file fragmentation. However, I agree
> things can be made more optimal.
> 
> So, with adding the above, there are *two* issues at play here:
> 1) Most/older versions of U-Boot not able to efficiently load files
> from ext4 filesystems, that cross multiple extents. I am aware of two
> fixes, see below.
> 2) WIC uses mkext4fs in such a way that files can cross multiple
> (allocation) extents. This is sub-optimal and know to cause issues
> with many U-Boot versions (deployed on existing systems in the field).
> The problems shows "randomly" with large files being deployed to the
> ext4 filesystem. We (OE/Yocto) may want to fix this.

That'd be best. One would expect that creating a filesystem "offline" would 
yield something optimal in terms of the way files are allocated, since 
everything about all files is already known when it's written.

My current workaround is to just mount the device and unpack the rootfs tar 
onto that, which yields a filesystem that u-boot can read at normal speed.


>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
> 
> Google for these two commits. I cherry-picked the first one for my
> project and can confirm it brings back the desired performance. (Still
> assuming this is the identical cause as for TO/Mike.)
> 
> commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> Author: Matt Madison <matt@madison.systems>
> Date:   Sat Aug 19 08:46:46 2017 -0700
> 
>      ext4: cache extent blocks during file reads
> 
>      A simpler and less-efficient approach to solving
>      the problem with extent index blocks than what
>      was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
>      but one that does not break write operations.
> 
>      Signed-off-by: Matt Madison <matt@madison.systems>

I'll give it a try...

You mentioned "two" commits?

Re: wic creates ext4 images that read really slow in u-boot

[Mike Looijmans]

>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.

Browsing through u-boot commits, I found this gem that's disabled by default:

CONFIG_BLOCK_CACHE=y

The commit is this one:

commit e40cf34a29f1b248643731a11fb1c6f0520d016c
Author: Eric Nelson <eric@nelint.com>
Date:   Mon Mar 28 10:05:44 2016 -0700

     drivers: block: add block device cache

     Add a block device cache to speed up repeated reads of block devices by
     various filesystems.

     This small amount of cache can dramatically speed up filesystem
     operations by skipping repeated reads of common areas of a block
     device (typically directory structures).

     This has shown to have some benefit on FAT filesystem operations of
     loading a kernel and RAM disk, but more dramatic benefits on ext4
     filesystems when the kernel and/or RAM disk are spread across
     multiple extent header structures as described in commit fc0fc50.
...

I'll be giving that a test drive soon...

Re: wic creates ext4 images that read really slow in u-boot

[Leon Woestenberg]

Hello Mike, all,

On Wed, Feb 20, 2019 at 11:42 AM Mike Looijmans <mike.looijmans@topic.nl> wrote:
>
> On 19-02-19 21:45, Leon Woestenberg wrote:
> > Hello all,
> >
> > On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
> >> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
> >>>
> >>> Hello Mike,
> >>>
> >>> sounds familiar.
> >>>
> >>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> >>>>
> >>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
> >>>> The issue turned out to be the ext4 partition created by wic.
> >>>
> >>> I suspect it's not WIC's fault.
> >>>>
> >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
> >
> > Google for these two commits. I cherry-picked the first one for my
> > project and can confirm it brings back the desired performance. (Still
> > assuming this is the identical cause as for TO/Mike.)
> >
> > commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> > Author: Matt Madison <matt@madison.systems>
> > Date:   Sat Aug 19 08:46:46 2017 -0700
> >
> >      ext4: cache extent blocks during file reads
> >
> >      A simpler and less-efficient approach to solving
> >      the problem with extent index blocks than what
> >      was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
> >      but one that does not break write operations.
> >
> >      Signed-off-by: Matt Madison <matt@madison.systems>
>
> I'll give it a try...
>
> You mentioned "two" commits?

Yes the commit above, and *another* approach referred to and mentioned
in the commit message (also with hash).

The block cache feature may solve the problem in a more generic way
(also caching things like directory lookups), whereas the ext4 extent
patches are more specific to ext4 only.

I found the ext4 fixes starting from here:
https://github.com/madisongh/meta-tegra/issues/42

Regards,

Leon.

Re: wic creates ext4 images that read really slow in u-boot

[Burton, Ross]

On Thu, 21 Feb 2019 at 10:17, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> That'd be best. One would expect that creating a filesystem "offline" would
> yield something optimal in terms of the way files are allocated, since
> everything about all files is already known when it's written.

But because we don't run image generation as root in a loopback mount,
we're limited to how clever e2fsprogs's debugfs is, *not* the kernel
driver.

Just a few weeks ago I fixed a bug in debugfs where it was only
writing the first 2^31 bits of a file...

Ross

Re: wic creates ext4 images that read really slow in u-boot

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Wed, Feb 20, 2019 at 10:55:20AM +0000, Mike Looijmans wrote:

> >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
> 
> Browsing through u-boot commits, I found this gem that's disabled by default:
> 
> CONFIG_BLOCK_CACHE=y
> 
> The commit is this one:
> 
> commit e40cf34a29f1b248643731a11fb1c6f0520d016c
> Author: Eric Nelson <eric@nelint.com>
> Date:   Mon Mar 28 10:05:44 2016 -0700
> 
>      drivers: block: add block device cache
> 
>      Add a block device cache to speed up repeated reads of block devices by
>      various filesystems.

For the record, it's not that it's disabled by default but rather to
enable it, you need to enable other "new" frameworks and so forth.  And
at least for "legacy" platforms, there's a certain level of pushback
against doing the work required to enable that (as to be honest, it is
more than just enable a few CONFIG options and re-test).  A
counter-pushback of "but it enables things that fix some bad performance
problems!" would be really helpful.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]