* Re: wic creates ext4 images that read really slow in u-boot
2019-02-19 20:45 ` Leon Woestenberg
@ 2019-02-20 9:12 ` Jack Mitchell
2019-02-20 10:42 ` Mike Looijmans
2019-02-20 10:55 ` Mike Looijmans
2 siblings, 0 replies; 10+ messages in thread
From: Jack Mitchell @ 2019-02-20 9:12 UTC (permalink / raw)
To: openembedded-core
On 19/02/2019 20:45, Leon Woestenberg wrote:
> Hello all,
>
> On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
>>>
>>> Hello Mike,
>>>
>>> sounds familiar.
>>>
>>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>>>
>>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
>>>> The issue turned out to be the ext4 partition created by wic.
>>>
>>> I suspect it's not WIC's fault.
>>>>
>>>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>>>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>>>
>>>> Now if I boot the board rename and copy that file onto itself, then it's
>>>> suddenly normal again when I reboot the board:
>>>>
>>>> ZynqMP> load mmc 0:2 0x10000
>>>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
>>>> creation tool that causes this?
>>>
>>> I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
>>>
>>> Copying the file makes the copy being layed out in one extend probably.
>>
>>
>> If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
>>
>> Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
>>
> I am not sure if "fragmented" is the right term in terms of filesystem
> details. Filesystem allocation extents (not "extends" as I mistyped
> earlier) are different from heavy file fragmentation. However, I agree
> things can be made more optimal.
>
> So, with adding the above, there are *two* issues at play here:
> 1) Most/older versions of U-Boot not able to efficiently load files
> from ext4 filesystems, that cross multiple extents. I am aware of two
> fixes, see below.
> 2) WIC uses mkext4fs in such a way that files can cross multiple
> (allocation) extents. This is sub-optimal and know to cause issues
> with many U-Boot versions (deployed on existing systems in the field).
> The problems shows "randomly" with large files being deployed to the
> ext4 filesystem. We (OE/Yocto) may want to fix this.
>
>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
>
> Google for these two commits. I cherry-picked the first one for my
> project and can confirm it brings back the desired performance. (Still
> assuming this is the identical cause as for TO/Mike.)
>
> commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> Author: Matt Madison <matt@madison.systems>
> Date: Sat Aug 19 08:46:46 2017 -0700
>
> ext4: cache extent blocks during file reads
>
> A simpler and less-efficient approach to solving
> the problem with extent index blocks than what
> was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
> but one that does not break write operations.
>
> Signed-off-by: Matt Madison <matt@madison.systems>
>
> Regards,
>
> Leon.
>
> p.s. excuse the earlier HTML mail with signature
>
I can concur that I have also used this patch to fix speed issues with
ext4 reads. The problem becomes particularly prominent when you have
large files on a small partition which get over-written. I'm sure I have
seen a patch go into uboot mainline recently which improved performance
but I can't find it or it never actually made it into the tree.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: wic creates ext4 images that read really slow in u-boot
2019-02-19 20:45 ` Leon Woestenberg
2019-02-20 9:12 ` Jack Mitchell
@ 2019-02-20 10:42 ` Mike Looijmans
2019-02-20 15:44 ` Leon Woestenberg
2019-02-21 10:43 ` Burton, Ross
2019-02-20 10:55 ` Mike Looijmans
2 siblings, 2 replies; 10+ messages in thread
From: Mike Looijmans @ 2019-02-20 10:42 UTC (permalink / raw)
To: Leon Woestenberg, Andre McCurdy; +Cc: openembedded-core
On 19-02-19 21:45, Leon Woestenberg wrote:
> Hello all,
>
> On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
>>>
>>> Hello Mike,
>>>
>>> sounds familiar.
>>>
>>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>>>
>>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
>>>> The issue turned out to be the ext4 partition created by wic.
>>>
>>> I suspect it's not WIC's fault.
>>>>
>>>> ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin
>>>> 19311092 bytes read in 85529 ms (219.7 KiB/s)
>>>>
>>>> Now if I boot the board rename and copy that file onto itself, then it's
>>>> suddenly normal again when I reboot the board:
>>>>
>>>> ZynqMP> load mmc 0:2 0x10000
>>>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
>>>> creation tool that causes this?
>>>
>>> I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
>>>
>>> Copying the file makes the copy being layed out in one extend probably.
>>
>>
>> If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
>>
>> Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
>>
> I am not sure if "fragmented" is the right term in terms of filesystem
> details. Filesystem allocation extents (not "extends" as I mistyped
> earlier) are different from heavy file fragmentation. However, I agree
> things can be made more optimal.
>
> So, with adding the above, there are *two* issues at play here:
> 1) Most/older versions of U-Boot not able to efficiently load files
> from ext4 filesystems, that cross multiple extents. I am aware of two
> fixes, see below.
> 2) WIC uses mkext4fs in such a way that files can cross multiple
> (allocation) extents. This is sub-optimal and know to cause issues
> with many U-Boot versions (deployed on existing systems in the field).
> The problems shows "randomly" with large files being deployed to the
> ext4 filesystem. We (OE/Yocto) may want to fix this.
That'd be best. One would expect that creating a filesystem "offline" would
yield something optimal in terms of the way files are allocated, since
everything about all files is already known when it's written.
My current workaround is to just mount the device and unpack the rootfs tar
onto that, which yields a filesystem that u-boot can read at normal speed.
>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
>
> Google for these two commits. I cherry-picked the first one for my
> project and can confirm it brings back the desired performance. (Still
> assuming this is the identical cause as for TO/Mike.)
>
> commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> Author: Matt Madison <matt@madison.systems>
> Date: Sat Aug 19 08:46:46 2017 -0700
>
> ext4: cache extent blocks during file reads
>
> A simpler and less-efficient approach to solving
> the problem with extent index blocks than what
> was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
> but one that does not break write operations.
>
> Signed-off-by: Matt Madison <matt@madison.systems>
I'll give it a try...
You mentioned "two" commits?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: wic creates ext4 images that read really slow in u-boot
2019-02-20 10:42 ` Mike Looijmans
@ 2019-02-20 15:44 ` Leon Woestenberg
2019-02-21 10:43 ` Burton, Ross
1 sibling, 0 replies; 10+ messages in thread
From: Leon Woestenberg @ 2019-02-20 15:44 UTC (permalink / raw)
To: Mike Looijmans; +Cc: openembedded-core
Hello Mike, all,
On Wed, Feb 20, 2019 at 11:42 AM Mike Looijmans <mike.looijmans@topic.nl> wrote:
>
> On 19-02-19 21:45, Leon Woestenberg wrote:
> > Hello all,
> >
> > On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy <armccurdy@gmail.com> wrote:
> >> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg <leon@sidebranch.com> wrote:
> >>>
> >>> Hello Mike,
> >>>
> >>> sounds familiar.
> >>>
> >>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> >>>>
> >>>> Took me a while to figure out why my board took 90 seconds to boot suddenly.
> >>>> The issue turned out to be the ext4 partition created by wic.
> >>>
> >>> I suspect it's not WIC's fault.
> >>>>
> >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
> >
> > Google for these two commits. I cherry-picked the first one for my
> > project and can confirm it brings back the desired performance. (Still
> > assuming this is the identical cause as for TO/Mike.)
> >
> > commit 855b8e4f9f255415a7753819e392e4b5d984f35c
> > Author: Matt Madison <matt@madison.systems>
> > Date: Sat Aug 19 08:46:46 2017 -0700
> >
> > ext4: cache extent blocks during file reads
> >
> > A simpler and less-efficient approach to solving
> > the problem with extent index blocks than what
> > was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5,
> > but one that does not break write operations.
> >
> > Signed-off-by: Matt Madison <matt@madison.systems>
>
> I'll give it a try...
>
> You mentioned "two" commits?
Yes the commit above, and *another* approach referred to and mentioned
in the commit message (also with hash).
The block cache feature may solve the problem in a more generic way
(also caching things like directory lookups), whereas the ext4 extent
patches are more specific to ext4 only.
I found the ext4 fixes starting from here:
https://github.com/madisongh/meta-tegra/issues/42
Regards,
Leon.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: wic creates ext4 images that read really slow in u-boot
2019-02-20 10:42 ` Mike Looijmans
2019-02-20 15:44 ` Leon Woestenberg
@ 2019-02-21 10:43 ` Burton, Ross
1 sibling, 0 replies; 10+ messages in thread
From: Burton, Ross @ 2019-02-21 10:43 UTC (permalink / raw)
To: Mike Looijmans; +Cc: openembedded-core
On Thu, 21 Feb 2019 at 10:17, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> That'd be best. One would expect that creating a filesystem "offline" would
> yield something optimal in terms of the way files are allocated, since
> everything about all files is already known when it's written.
But because we don't run image generation as root in a loopback mount,
we're limited to how clever e2fsprogs's debugfs is, *not* the kernel
driver.
Just a few weeks ago I fixed a bug in debugfs where it was only
writing the first 2^31 bits of a file...
Ross
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: wic creates ext4 images that read really slow in u-boot
2019-02-19 20:45 ` Leon Woestenberg
2019-02-20 9:12 ` Jack Mitchell
2019-02-20 10:42 ` Mike Looijmans
@ 2019-02-20 10:55 ` Mike Looijmans
2019-02-21 14:33 ` Tom Rini
2 siblings, 1 reply; 10+ messages in thread
From: Mike Looijmans @ 2019-02-20 10:55 UTC (permalink / raw)
To: Leon Woestenberg, Andre McCurdy; +Cc: openembedded-core
>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Browsing through u-boot commits, I found this gem that's disabled by default:
CONFIG_BLOCK_CACHE=y
The commit is this one:
commit e40cf34a29f1b248643731a11fb1c6f0520d016c
Author: Eric Nelson <eric@nelint.com>
Date: Mon Mar 28 10:05:44 2016 -0700
drivers: block: add block device cache
Add a block device cache to speed up repeated reads of block devices by
various filesystems.
This small amount of cache can dramatically speed up filesystem
operations by skipping repeated reads of common areas of a block
device (typically directory structures).
This has shown to have some benefit on FAT filesystem operations of
loading a kernel and RAM disk, but more dramatic benefits on ext4
filesystems when the kernel and/or RAM disk are spread across
multiple extent header structures as described in commit fc0fc50.
...
I'll be giving that a test drive soon...
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: wic creates ext4 images that read really slow in u-boot
2019-02-20 10:55 ` Mike Looijmans
@ 2019-02-21 14:33 ` Tom Rini
0 siblings, 0 replies; 10+ messages in thread
From: Tom Rini @ 2019-02-21 14:33 UTC (permalink / raw)
To: Mike Looijmans; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]
On Wed, Feb 20, 2019 at 10:55:20AM +0000, Mike Looijmans wrote:
> >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
>
> Browsing through u-boot commits, I found this gem that's disabled by default:
>
> CONFIG_BLOCK_CACHE=y
>
> The commit is this one:
>
> commit e40cf34a29f1b248643731a11fb1c6f0520d016c
> Author: Eric Nelson <eric@nelint.com>
> Date: Mon Mar 28 10:05:44 2016 -0700
>
> drivers: block: add block device cache
>
> Add a block device cache to speed up repeated reads of block devices by
> various filesystems.
For the record, it's not that it's disabled by default but rather to
enable it, you need to enable other "new" frameworks and so forth. And
at least for "legacy" platforms, there's a certain level of pushback
against doing the work required to enable that (as to be honest, it is
more than just enable a few CONFIG options and re-test). A
counter-pushback of "but it enables things that fix some bad performance
problems!" would be really helpful.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread