RFC: Devices for files and partitions

RFC: Devices for files and partitions

[Simon Glass]

Hi,

At present U-Boot avoids the concept of 'opening' a file. Being in a
bootloader environment, it is normally better to take the action
immediately and avoid any caching, for example, since there is no
background task to clean up afterwards.

Having said that, the concept of a file is quite useful, for example
to write the output of a command to a file, or to open a file and read
it a line at a time.

Another case has come to light in that EFI wants to access files using
a file handle. This currently uses parallel data structures and does
not map very well in U-Boot.

Finally, partitions has a similar issue, where defining them as a
device can have benefits, e.g. to specify the device to use directly,
rather than with the 'type dev:part' approach, perhaps providing them
in the devicetree, etc.

For the above reasons, I propose that we implement, as an option,
support for files and partitions within driver model.

Regards,
Simon

Re: RFC: Devices for files and partitions

[AKASHI Takahiro]

On Tue, Jul 06, 2021 at 05:05:19PM -0600, Simon Glass wrote:
> Hi,
> 
> At present U-Boot avoids the concept of 'opening' a file. Being in a
> bootloader environment, it is normally better to take the action
> immediately and avoid any caching, for example, since there is no
> background task to clean up afterwards.
> 
> Having said that, the concept of a file is quite useful, for example
> to write the output of a command to a file, or to open a file and read
> it a line at a time.
> 
> Another case has come to light in that EFI wants to access files using
> a file handle. This currently uses parallel data structures and does
> not map very well in U-Boot.
> 
> Finally, partitions has a similar issue, where defining them as a
> device can have benefits, e.g. to specify the device to use directly,
> rather than with the 'type dev:part' approach, perhaps providing them
> in the devicetree, etc.
> 
> For the above reasons, I propose that we implement, as an option,
> support for files and partitions within driver model.

+1

# Nobody has commented yet :)

Regarding a "file (or file descriptor)", we have already implemented
the same concept in efi_loader. So technically, it won't be a hard-work.
Regarding "partitions as udevice," I have posted an experimental
patch [1]. So it must also be feasible.

One of my concerns is what benefit end users may enjoy.

-Takahiro Akashi

[1] https://lists.denx.de/pipermail/u-boot/2019-February/357937.html
    https://lists.denx.de/pipermail/u-boot/2019-February/357934.html

> Regards,
> Simon

Re: RFC: Devices for files and partitions

[Heinrich Schuchardt]

On 7/9/21 4:44 AM, AKASHI Takahiro wrote:
> On Tue, Jul 06, 2021 at 05:05:19PM -0600, Simon Glass wrote:
>> Hi,
>>
>> At present U-Boot avoids the concept of 'opening' a file. Being in a
>> bootloader environment, it is normally better to take the action
>> immediately and avoid any caching, for example, since there is no
>> background task to clean up afterwards.
>>
>> Having said that, the concept of a file is quite useful, for example
>> to write the output of a command to a file, or to open a file and read
>> it a line at a time.
>>
>> Another case has come to light in that EFI wants to access files using
>> a file handle. This currently uses parallel data structures and does
>> not map very well in U-Boot.

Not having a file descriptor slows down file operation because for each
individual access to a file we

* read the partition table
* look up the file in the directory structure

If you only use the 'load' command, this does not matter because there
is only a single read. But in the UEFI sub-system we use multiple API calls:

* open the volume
* open the file
* read the file size (for buffer allocation)
* read the file

The file descriptors in the UEFI sub-system only hold device, partition
ID, file path.

>>
>> Finally, partitions has a similar issue, where defining them as a
>> device can have benefits, e.g. to specify the device to use directly,
>> rather than with the 'type dev:part' approach, perhaps providing them
>> in the devicetree, etc.
>>
>> For the above reasons, I propose that we implement, as an option,
>> support for files and partitions within driver model.

With partitions as devices we will have to read the partition table only
once when the block device is probed. - For the UEFI sub-system we need
all block devices to be probed before calling the UEFI binary (e.g. GRUB).

>
> +1
>
> # Nobody has commented yet :)
>
> Regarding a "file (or file descriptor)", we have already implemented
> the same concept in efi_loader. So technically, it won't be a hard-work.
> Regarding "partitions as udevice," I have posted an experimental
> patch [1]. So it must also be feasible.

Our UEFI file descriptor is not helpful because it does not buffer the
relevant data to speed up file access. E.g. for the FAT driver a file
descriptor could buffer some information from the directory entry (file
attributes, dates) and some from the FAT table (assigned extends).

>
> One of my concerns is what benefit end users may enjoy.

Improved performance.

Best regards

Heinrich

>
> -Takahiro Akashi
>
> [1] https://lists.denx.de/pipermail/u-boot/2019-February/357937.html
>      https://lists.denx.de/pipermail/u-boot/2019-February/357934.html
>
>> Regards,
>> Simon

Re: RFC: Devices for files and partitions

[Simon Glass]

Hi,

On Thu, 8 Jul 2021 at 23:33, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 7/9/21 4:44 AM, AKASHI Takahiro wrote:
> > On Tue, Jul 06, 2021 at 05:05:19PM -0600, Simon Glass wrote:
> >> Hi,
> >>
> >> At present U-Boot avoids the concept of 'opening' a file. Being in a
> >> bootloader environment, it is normally better to take the action
> >> immediately and avoid any caching, for example, since there is no
> >> background task to clean up afterwards.
> >>
> >> Having said that, the concept of a file is quite useful, for example
> >> to write the output of a command to a file, or to open a file and read
> >> it a line at a time.
> >>
> >> Another case has come to light in that EFI wants to access files using
> >> a file handle. This currently uses parallel data structures and does
> >> not map very well in U-Boot.
>
> Not having a file descriptor slows down file operation because for each
> individual access to a file we
>
> * read the partition table
> * look up the file in the directory structure
>
> If you only use the 'load' command, this does not matter because there
> is only a single read. But in the UEFI sub-system we use multiple API calls:
>
> * open the volume
> * open the file
> * read the file size (for buffer allocation)
> * read the file
>
> The file descriptors in the UEFI sub-system only hold device, partition
> ID, file path.

Arguably that is a potential benefit, dependent on us implementing
(write-through?) caching of this info.

>
> >>
> >> Finally, partitions has a similar issue, where defining them as a
> >> device can have benefits, e.g. to specify the device to use directly,
> >> rather than with the 'type dev:part' approach, perhaps providing them
> >> in the devicetree, etc.
> >>
> >> For the above reasons, I propose that we implement, as an option,
> >> support for files and partitions within driver model.
>
> With partitions as devices we will have to read the partition table only
> once when the block device is probed. - For the UEFI sub-system we need
> all block devices to be probed before calling the UEFI binary (e.g. GRUB).

It would be better if UEFI could support lazy probing, but that's
off-topic here.

>
> >
> > +1
> >
> > # Nobody has commented yet :)

Thank you for commenting.

> >
> > Regarding a "file (or file descriptor)", we have already implemented
> > the same concept in efi_loader. So technically, it won't be a hard-work.
> > Regarding "partitions as udevice," I have posted an experimental
> > patch [1]. So it must also be feasible.
>
> Our UEFI file descriptor is not helpful because it does not buffer the
> relevant data to speed up file access. E.g. for the FAT driver a file
> descriptor could buffer some information from the directory entry (file
> attributes, dates) and some from the FAT table (assigned extends).
>
> >
> > One of my concerns is what benefit end users may enjoy.
>
> Improved performance.

With a bit of effort, yes, with UEFI at least, by the sound of it.

I'm not sure how much difference it would make for non-EFI use though.

To me this is an internal thing, a bit like driver model, where the
benefit is easier development and scripting, which might benefit end
users indirectly through faster time-to-market and fewer bugs?

Takahiro, since you have already started this work, do you want to
continue it? I think the original patch needed a test.

>
> Best regards
>
> Heinrich
>
> >
> > -Takahiro Akashi
> >
> > [1] https://lists.denx.de/pipermail/u-boot/2019-February/357937.html
> >      https://lists.denx.de/pipermail/u-boot/2019-February/357934.html
> >
> >> Regards,
> >> Simon

Regards,
Simon

>