Feature request: Adding tools to be able to split the linux-firmware package into separate components

Feature request: Adding tools to be able to split the linux-firmware package into separate components

[Azamat Hackimov]

The size of linux-firmware already reaches over 1 GB. Linux
distributions typically do not split the linux-firmware package into
components, which results in the user's installation storing files
that are never used by the system and take up unnecessary space. This
problem is especially acute for embedded systems or systems running in
containers.

There has been a request among some distributions to separate
linux-firmware into components in order to make it easier for the user
to install and use firmware. Among them are Arch Linux[1], Fedora[2],
Alpine[3], however, separation on the side of the maintainers of the
distribution is fraught with difficulties associated with the
inability to reliably separate the individual components of the
linux-firmware package by driver or even by category.

I propose to provide a similar mechanism for filtering prerequisites
on the side of the linux-firmware package. To do this, I propose to
transform the WHENCE file into a structured file that can be processed
by an additional tool that can filter the necessary components by
driver, category, manufacturer or license.

Suggested structured file format is YAML (although other formats are
possible), example entry (actual existing entry from WHENCE):

```
drivers:
  - driver: emi62
    description: EMI 6|2m USB Audio interface
    category:
      - drivers/usb/misc
    vendor: Emagic
    license: Unknown
    info: |
      Converted from Intel HEX files, used in our binary representation of ihex.
    files:
      - name: emi62/bitstream.fw
        version: 1.0.0.191
        info: DATE=2002oct28
      - name: emi62/loader.fw
        version: 1.0.2.002
        info: DATE=10.01.2002
      - name: emi62/midi.fw
        version: 1.04.062
        info: DATE=16.10.2002
      - name: emi62/spdif.fw
        version: 1.04.062
        info: DATE=16.10.2002
```

The firmware-install utility to query and filter a structured file
should parse the following options:

firmware-install list [--category <LIST>] [--driver <LIST>] [--license
<LIST>] [--vendor <LIST>]
  List drivers and firmwares that applies to filter

firmware-install show <DRIVER>
  Show information about driver

firmware-install install <DESTINATION> [--category <LIST>] [--driver
<LIST>] [--license <LIST>] [--vendor <LIST>]
  Install firmwares that applies to filter

firmware-install check
  Check format and content of structured file (like check_whence.py does)

After the implementation of such a utility, maintainers of Linux
distributions will have a convenient tool that will allow them to
create sets of linux-firmware-* packages with content separation
instead of a monolithic installation of all firmware.

[1] https://bugs.archlinux.org/task/72559
[2] https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
[3] https://lists.alpinelinux.org/~alpine/devel/%3C4d19a51c-3eba-9775-0208-4d87da66effc%40bitmessage.ch%3E

Re: Feature request: Adding tools to be able to split the linux-firmware package into separate components

[Josh Boyer]

On Wed, Oct 19, 2022 at 12:15 PM Azamat Hackimov
<azamat.hackimov@gmail.com> wrote:
>
> The size of linux-firmware already reaches over 1 GB. Linux
> distributions typically do not split the linux-firmware package into
> components, which results in the user's installation storing files
> that are never used by the system and take up unnecessary space. This
> problem is especially acute for embedded systems or systems running in
> containers.

Why is firmware installed in containers at all?

> There has been a request among some distributions to separate
> linux-firmware into components in order to make it easier for the user
> to install and use firmware. Among them are Arch Linux[1], Fedora[2],
> Alpine[3], however, separation on the side of the maintainers of the
> distribution is fraught with difficulties associated with the
> inability to reliably separate the individual components of the
> linux-firmware package by driver or even by category.
>
> I propose to provide a similar mechanism for filtering prerequisites
> on the side of the linux-firmware package. To do this, I propose to
> transform the WHENCE file into a structured file that can be processed
> by an additional tool that can filter the necessary components by
> driver, category, manufacturer or license.

If someone is going to implement this, I'd suggest just creating a new
file to start with, and keeping WHENCE as-is.

> Suggested structured file format is YAML (although other formats are
> possible), example entry (actual existing entry from WHENCE):
>
> ```
> drivers:
>   - driver: emi62
>     description: EMI 6|2m USB Audio interface
>     category:
>       - drivers/usb/misc
>     vendor: Emagic
>     license: Unknown
>     info: |
>       Converted from Intel HEX files, used in our binary representation of ihex.
>     files:
>       - name: emi62/bitstream.fw
>         version: 1.0.0.191
>         info: DATE=2002oct28
>       - name: emi62/loader.fw
>         version: 1.0.2.002
>         info: DATE=10.01.2002
>       - name: emi62/midi.fw
>         version: 1.04.062
>         info: DATE=16.10.2002
>       - name: emi62/spdif.fw
>         version: 1.04.062
>         info: DATE=16.10.2002
> ```
>
> The firmware-install utility to query and filter a structured file
> should parse the following options:
>
> firmware-install list [--category <LIST>] [--driver <LIST>] [--license
> <LIST>] [--vendor <LIST>]
>   List drivers and firmwares that applies to filter

This implies the linux-firmware project and utilities would have to
monitor Linux kernel driver information.  I don't think we want to tie
the two directly together like that.  There's little need from the
firmware side because the kernel drivers already have the firmware
they need embedded in the module info.

> firmware-install show <DRIVER>
>   Show information about driver

This is already covered by modinfo.

> firmware-install install <DESTINATION> [--category <LIST>] [--driver
> <LIST>] [--license <LIST>] [--vendor <LIST>]
>   Install firmwares that applies to filter

This would be useful for highly specific, non-changing installations
(e.g. embedded as you mention).  However, a distribution is not going
to use this as they have to install firmware for any kind of device a
user may have.  They can already package the files in separate
distribution packages if they wish.

> firmware-install check
>   Check format and content of structured file (like check_whence.py does)
>
> After the implementation of such a utility, maintainers of Linux
> distributions will have a convenient tool that will allow them to
> create sets of linux-firmware-* packages with content separation
> instead of a monolithic installation of all firmware.

They can and do already do this.

I'm not against the overall idea, but I don't have time to implement
anything for this and it's unclear to me what the overall value would
be.  If someone wants to send patches for this, I'd happily review
them.

josh

> [1] https://bugs.archlinux.org/task/72559
> [2] https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
> [3] https://lists.alpinelinux.org/~alpine/devel/%3C4d19a51c-3eba-9775-0208-4d87da66effc%40bitmessage.ch%3E

Re: Feature request: Adding tools to be able to split the linux-firmware package into separate components

[Azamat Hackimov]

Hello again.

Josh, thank you for your feedback. The main idea for the new feature
was to create a tool for more efficient control for obtaining
information and selective installation of project files. I created a
feature request to get some feedback and general impression of the
idea and planned to implement it myself. So, based on the feedback, I
implemented the necessary changes and published them in the
https://github.com/winterheart/linux-firmware repository (yaml
branch).

I will briefly list the new features. All functionality is
concentrated in firmware-install.py and WHENCE.yaml files. The
functionality is implemented in such a way that it does not interfere
with the existing check_whence.py and copy-firmware.sh in any way
(that is, you can use either one or that without losing
functionality). firmware-install.py requires Python 3 (tested with
Python 3.10) and the pyyaml (to load WHENCE.yaml) and marshmallow (to
deserialize and validate the processed data) packages to run. All
packages are available in the repositories of most Linux
distributions, for example on Debian / Ubuntu they can be installed as
follows:

apt-get install python-marshmallow pyyaml

firmware-install.py supports the following types of commands (run
`./firmware-install.py -h` and `./firmware-install.py COMMAND -h` for
additional info):

* check - fully checks the contents of the repository in a way similar
to check_whence.py, and also validates the schema and contents of
WHENCE.yaml
* info - displays information on records according to the specified
selection criteria
* install - installs to the specified directory according to the
specified selection criteria
* list - displays a list of possible values for entries, vendors,
categories or licenses

The install command without additional options will perform a full
installation in exactly the same way as copy-firmware.sh. The contents
of the installation directories in this case will be identical.

Work example of info subcommand:

./firmware-install.py info -n bnx2
Entry: bnx2
Description: Broadcom NetXtremeII
Categories:
  - drivers/net/ethernet
Vendor: Broadcom
License:
  Name: Redistributable
  Copyright: Copyright (c) 2004-2010 Broadcom Corporation
  Info: This file contains firmware data derived from proprietary unpublished
source code, Copyright (c) 2004 - 2010 Broadcom Corporation.

Permission is hereby granted for the distribution of this firmware data
in hexadecimal or equivalent format, provided this copyright notice is
accompanying it.

Info:
Found in hex form in kernel source.
Size: 215,144 bytes
Files:
  - bnx2/bnx2-mips-06-6.2.3.fw
  - bnx2/bnx2-mips-09-6.2.1b.fw
  - bnx2/bnx2-rv2p-06-6.0.15.fw
  - bnx2/bnx2-rv2p-09-6.0.17.fw
  - bnx2/bnx2-rv2p-09ax-6.0.17.fw
Links:
None
--------

The WHENCE.yaml format is detailed in the updated README and allows
you to enter any information. Content of WHENCE.yaml currently
synchronized with last commit in main branch of linux-firmware
repository (commit 8bb7562).

Please take a look and try new functionality. If you have any
additional questions, I will be happy to answer them.

-- 
From Siberia with Love!

Re: Feature request: Adding tools to be able to split the linux-firmware package into separate components

[Azamat Hackimov]

Hello.

Is there any feedback or suggestions?

вс, 6 нояб. 2022 г. в 20:53, Azamat Hackimov <azamat.hackimov@gmail.com>:
>
> Hello again.
>
> Josh, thank you for your feedback. The main idea for the new feature
> was to create a tool for more efficient control for obtaining
> information and selective installation of project files. I created a
> feature request to get some feedback and general impression of the
> idea and planned to implement it myself. So, based on the feedback, I
> implemented the necessary changes and published them in the
> https://github.com/winterheart/linux-firmware repository (yaml
> branch).
>
> I will briefly list the new features. All functionality is
> concentrated in firmware-install.py and WHENCE.yaml files. The
> functionality is implemented in such a way that it does not interfere
> with the existing check_whence.py and copy-firmware.sh in any way
> (that is, you can use either one or that without losing
> functionality). firmware-install.py requires Python 3 (tested with
> Python 3.10) and the pyyaml (to load WHENCE.yaml) and marshmallow (to
> deserialize and validate the processed data) packages to run. All
> packages are available in the repositories of most Linux
> distributions, for example on Debian / Ubuntu they can be installed as
> follows:
>
> apt-get install python-marshmallow pyyaml
>
> firmware-install.py supports the following types of commands (run
> `./firmware-install.py -h` and `./firmware-install.py COMMAND -h` for
> additional info):
>
> * check - fully checks the contents of the repository in a way similar
> to check_whence.py, and also validates the schema and contents of
> WHENCE.yaml
> * info - displays information on records according to the specified
> selection criteria
> * install - installs to the specified directory according to the
> specified selection criteria
> * list - displays a list of possible values for entries, vendors,
> categories or licenses
>
> The install command without additional options will perform a full
> installation in exactly the same way as copy-firmware.sh. The contents
> of the installation directories in this case will be identical.
>
> Work example of info subcommand:
>
> ./firmware-install.py info -n bnx2
> Entry: bnx2
> Description: Broadcom NetXtremeII
> Categories:
>   - drivers/net/ethernet
> Vendor: Broadcom
> License:
>   Name: Redistributable
>   Copyright: Copyright (c) 2004-2010 Broadcom Corporation
>   Info: This file contains firmware data derived from proprietary unpublished
> source code, Copyright (c) 2004 - 2010 Broadcom Corporation.
>
> Permission is hereby granted for the distribution of this firmware data
> in hexadecimal or equivalent format, provided this copyright notice is
> accompanying it.
>
> Info:
> Found in hex form in kernel source.
> Size: 215,144 bytes
> Files:
>   - bnx2/bnx2-mips-06-6.2.3.fw
>   - bnx2/bnx2-mips-09-6.2.1b.fw
>   - bnx2/bnx2-rv2p-06-6.0.15.fw
>   - bnx2/bnx2-rv2p-09-6.0.17.fw
>   - bnx2/bnx2-rv2p-09ax-6.0.17.fw
> Links:
> None
> --------
>
> The WHENCE.yaml format is detailed in the updated README and allows
> you to enter any information. Content of WHENCE.yaml currently
> synchronized with last commit in main branch of linux-firmware
> repository (commit 8bb7562).
>
> Please take a look and try new functionality. If you have any
> additional questions, I will be happy to answer them.
>
> --
> From Siberia with Love!



-- 
From Siberia with Love!