[Buildroot] [PATCH-NEXT v2 0/4] package/linux-firmware: install into images for early loading support

[Buildroot] [PATCH-NEXT v2 0/4] package/linux-firmware: install into images for early loading support

[Peter Korsgaard]

Some drivers request their firmware very early when built into the
kernel, even before the initramfs is mounted - So the only way to
provide firmware for those drivers is to include them directly in the
kernel with the CONFIG_EXTRA_FIRMWARE option.

To support this, let linux-firmware install its files into
BINARIES_DIR in addition to TARGET_DIR, similar to how we do it for
intel-microcode.

Peter Korsgaard (3):
  package/linux-firmware: make install logic macro accept a destination
    parameter
  package/linux-firmware: also install into images for early loading
    support
  linux: build after linux-firmware if enabled for early loading support

Yann E. MORIN (1):
  package/linux-firmware: rationalise install step

 linux/linux.mk                           |  3 ++-
 package/linux-firmware/linux-firmware.mk | 34 +++++++++---------------
 2 files changed, 15 insertions(+), 22 deletions(-)

-- 
2.20.1

[Buildroot] [PATCH-NEXT v2 1/4] package/linux-firmware: rationalise install step

[Peter Korsgaard]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

The logic we have for the installation of the firmware files is, to say
the least, non conventional. It is split in two parts:

  - one that copies files via an intermediate tarball: the tarball
    creation is used to detect if firmware files are missing (i.e. on
    a version bump) and fail the build if so, while the tarball
    extraction is the actual firmware installation;

  - one that copies directories one by one in a loop, removing the
    destination before the copy, to maintain a proper layout.

Needless to say, this is not very clean. First, there is no reason why
the directories can not be copied with the same mechanism as the files
themselves; not sure what I had in mind with b55bd5a9e25e...

Second, we're soon going to need the same installation step to copy the
firmware files in the images/ directory, to ease embedding in the kernel
image.

Rationalise this installation procedure.

Cherry-picking files and directories with cp, while still maintaining
the directory layout, is not trivial; rsync is not one of our
pre-requisites. So we're left with tar, which makes it easy. So we keep
using an intermediate tarball, but we use it for both files and
directories, and we generate it at build time, not install time.

That archive is then extracted during the installation.

Now the installation complexity is mostly located in the creation of the
symlinks, so we merge all of that directly into the _INSTALL_TARGET_CMDS
and drop the intermediate macros that have no longer any reason to exist.

This will also make it pretty simple to later install in the images/
directory.

Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
Changes since v1:
- New patch

package/linux-firmware/linux-firmware.mk | 29 +++++-------------------
 1 file changed, 6 insertions(+), 23 deletions(-)

diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index f23da171c6..9bc59435ef 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -638,22 +638,10 @@ LINUX_FIRMWARE_FILES += ti_3410.fw ti_5052.fw \
 LINUX_FIRMWARE_ALL_LICENSE_FILES += LICENCE.moxa
 endif
 
-ifneq ($(LINUX_FIRMWARE_FILES),)
-define LINUX_FIRMWARE_INSTALL_FILES
+ifneq ($(LINUX_FIRMWARE_FILES)$(LINUX_FIRMWARE_DIRS),)
+define LINUX_FIRMWARE_BUILD_CMDS
 	cd $(@D) && \
-		$(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \
-		$(TAR) xf install.tar -C $(TARGET_DIR)/lib/firmware
-endef
-endif
-
-ifneq ($(LINUX_FIRMWARE_DIRS),)
-# We need to rm-rf the destination directory to avoid copying
-# into it in itself, should we re-install the package.
-define LINUX_FIRMWARE_INSTALL_DIRS
-	$(foreach d,$(LINUX_FIRMWARE_DIRS), \
-		rm -rf $(TARGET_DIR)/lib/firmware/$(d); \
-		mkdir -p $(dir $(TARGET_DIR)/lib/firmware/$(d)); \
-		cp -a $(@D)/$(d) $(TARGET_DIR)/lib/firmware/$(d)$(sep))
+	$(TAR) cf br-firmware.tar $(sort $(LINUX_FIRMWARE_FILES) $(LINUX_FIRMWARE_DIRS))
 endef
 endif
 
@@ -686,7 +674,9 @@ endif
 # sure we canonicalize the pointed-to file, to cover the symlinks of the form
 # a/foo -> ../b/foo  where a/ (the directory where to put the symlink) does
 # not yet exist.
-define LINUX_FIRMWARE_CREATE_SYMLINKS
+define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
+	mkdir -p $(TARGET_DIR)/lib/firmware
+	$(TAR) xf $(@D)/br-firmware.tar -C $(TARGET_DIR)/lib/firmware/
 	cd $(TARGET_DIR)/lib/firmware/ ; \
 	sed -r -e '/^Link: (.+) -> (.+)$$/!d; s//\1 \2/' $(@D)/WHENCE | \
 	while read f d; do \
@@ -697,11 +687,4 @@ define LINUX_FIRMWARE_CREATE_SYMLINKS
 	done
 endef
 
-define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
-	mkdir -p $(TARGET_DIR)/lib/firmware
-	$(LINUX_FIRMWARE_INSTALL_FILES)
-	$(LINUX_FIRMWARE_INSTALL_DIRS)
-	$(LINUX_FIRMWARE_CREATE_SYMLINKS)
-endef
-
 $(eval $(generic-package))
-- 
2.20.1

[Buildroot] [PATCH-NEXT v2 2/4] package/linux-firmware: make install logic macro accept a destination parameter

[Peter Korsgaard]

So it can be reused for also installing into the images directory.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
Changes since v1:
- Reworked after patch 1 changed install logic

package/linux-firmware/linux-firmware.mk | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index 9bc59435ef..8defedd309 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -674,10 +674,10 @@ endif
 # sure we canonicalize the pointed-to file, to cover the symlinks of the form
 # a/foo -> ../b/foo  where a/ (the directory where to put the symlink) does
 # not yet exist.
-define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
-	mkdir -p $(TARGET_DIR)/lib/firmware
-	$(TAR) xf $(@D)/br-firmware.tar -C $(TARGET_DIR)/lib/firmware/
-	cd $(TARGET_DIR)/lib/firmware/ ; \
+define LINUX_FIRMWARE_INSTALL_FW
+	mkdir -p $(1)
+	$(TAR) xf $(@D)/br-firmware.tar -C $(1)
+	cd $(1) ; \
 	sed -r -e '/^Link: (.+) -> (.+)$$/!d; s//\1 \2/' $(@D)/WHENCE | \
 	while read f d; do \
 		if test -f $$(readlink -m $$(dirname $$f)/$$d); then \
@@ -687,4 +687,8 @@ define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
 	done
 endef
 
+define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
+	$(call LINUX_FIRMWARE_INSTALL_FW, $(TARGET_DIR)/lib/firmware)
+endef
+
 $(eval $(generic-package))
-- 
2.20.1

[Buildroot] [PATCH-NEXT v2 3/4] package/linux-firmware: also install into images for early loading support

[Peter Korsgaard]

Some drivers request their firmware very early when built into the kernel,
even before the initramfs is mounted - So the only way to provide firmware
for those drivers is to include them directly in the kernel with the
CONFIG_EXTRA_FIRMWARE option.

An example of this is the uC firmware for modern Intel GPUs.

Conceptually you can point CONFIG_EXTRA_FIRMWARE to
${TARGET_DIR}/lib/firmware, but then you cannot remove the firmware from the
initramfs and pay the size cost twice (inside the kernel + in initramfs), so
instead also install linux-firmware to the images dir, similar to how we do
it for intel-microcode.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
Changes since v1:
- Reworked after patch 1 changed install logic

package/linux-firmware/linux-firmware.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index 8defedd309..5e92b0446e 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -7,6 +7,7 @@
 LINUX_FIRMWARE_VERSION = 20201022
 LINUX_FIRMWARE_SITE = http://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
 LINUX_FIRMWARE_SITE_METHOD = git
+LINUX_FIRMWARE_INSTALL_IMAGES = YES
 
 LINUX_FIRMWARE_CPE_ID_VENDOR = kernel
 
@@ -691,4 +692,8 @@ define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
 	$(call LINUX_FIRMWARE_INSTALL_FW, $(TARGET_DIR)/lib/firmware)
 endef
 
+define LINUX_FIRMWARE_INSTALL_IMAGES_CMDS
+	$(call LINUX_FIRMWARE_INSTALL_FW, $(BINARIES_DIR))
+endef
+
 $(eval $(generic-package))
-- 
2.20.1

[Buildroot] [PATCH-NEXT v2 4/4] linux: build after linux-firmware if enabled for early loading support

[Peter Korsgaard]

To support building in (a subset of) the linux-firmware files into the
kernel using the CONFIG_EXTRA_FIRMWARE option, we need to ensure that the
firmware files are installed before the Linux kernel is built, similar to
how it is done for intel-microcode.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 248f47cdd407ba52a851d50ca29bd7fb2d0a9353)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
Changes since v1:
- No changes

 linux/linux.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index a212f42c28..5e4b319cf1 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -78,7 +78,8 @@ LINUX_MAKE_ENV = \
 
 LINUX_INSTALL_IMAGES = YES
 LINUX_DEPENDENCIES = host-kmod \
-	$(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode)
+	$(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) \
+	$(if $(BR2_PACKAGE_LINUX_FIRMWARE),linux-firmware)
 
 # Starting with 4.16, the generated kconfig paser code is no longer
 # shipped with the kernel sources, so we need flex and bison, but
-- 
2.20.1

[Buildroot] [PATCH-NEXT v2 0/4] package/linux-firmware: install into images for early loading support

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

 > Some drivers request their firmware very early when built into the
 > kernel, even before the initramfs is mounted - So the only way to
 > provide firmware for those drivers is to include them directly in the
 > kernel with the CONFIG_EXTRA_FIRMWARE option.

 > To support this, let linux-firmware install its files into
 > BINARIES_DIR in addition to TARGET_DIR, similar to how we do it for
 > intel-microcode.

 > Peter Korsgaard (3):
 >   package/linux-firmware: make install logic macro accept a destination
 >     parameter
 >   package/linux-firmware: also install into images for early loading
 >     support
 >   linux: build after linux-firmware if enabled for early loading support

 > Yann E. MORIN (1):
 >   package/linux-firmware: rationalise install step

Committed to next, thanks.

-- 
Bye, Peter Korsgaard