* [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
@ 2023-02-18 18:46 Emil Velikov
2023-02-18 18:46 ` [PATCH 01/15] WHENCE: remove trailing white space Emil Velikov
` (15 more replies)
0 siblings, 16 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Greetings everyone,
For a while now, the kernel has been accepting compressed firmware - be that xz
or zstd. There is a patch for the former on the ML, yet the latter is still
lacking.
While working through I've noticed a few issues - of various severity - so
decided to fix them all.
Mind you, I'm sending everything as single patch series to gauge feedback. I'm
quite happy to split and rework as per the community's feedback.
In a gist:
- remove quoted filenames in WHENCE - escape white space as needed
- remove _nearly_ all duplicate entries in WHENCE && add check
- NOTE: one entry remains, which I'm not sure how to resolve cleanly
- check if directory is listed as File - had a mistake like that locally
- check we have no symlinks and drop symlink workarounds
- quote deskdir (genuine bug) + few more shellcheck inspired fixes
- support xz and zstd compressed firmware - no intermediate files, pipe
directly into destdir
Kind regards
Emil
Emil Velikov (15):
WHENCE: remove trailing white space
WHENCE: remove unnecessary filename quotation
check_whence, WHENCE, copy-firmware: escape filenames with spaces
WHENCE: remove duplicate File entries
check_whence: error on duplicate file entries
check_whence: error on directory listed as File
copy-firmware: remove non-applicable file presence test
check_whence: error if File: is actually a link
check_whence: error if symlinks are in-tree
copy-firmware: remove unreachable symlink workarounds
copy-firmware: quote deskdir to prevent word splitting
copy-firmware: tweak sed invocation
copy-firmware: quote the output of dirname
copy-firmware: silence the last shellcheck warnings
Makefile, copy-firmware: support xz/zstd compressed firmware
Makefile | 8 ++++
WHENCE | 97 +++++++++++++++++++-----------------------------
check_whence.py | 44 +++++++++++++++++++---
copy-firmware.sh | 71 +++++++++++++++++------------------
4 files changed, 119 insertions(+), 101 deletions(-)
--
2.39.2
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 01/15] WHENCE: remove trailing white space
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 02/15] WHENCE: remove unnecessary filename quotation Emil Velikov
` (14 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
WHENCE | 36 ++++++++++++++++++------------------
1 file changed, 18 insertions(+), 18 deletions(-)
diff --git a/WHENCE b/WHENCE
index b6199d6..23c8847 100644
--- a/WHENCE
+++ b/WHENCE
@@ -395,7 +395,7 @@ Version: 8.07.00 MIDQ
Licence: Redistributable. See LICENCE.qla2xxx for details
-Available from http://ldriver.qlogic.com/firmware/
+Available from http://ldriver.qlogic.com/firmware/
--------------------------------------------------------------------------
@@ -2563,29 +2563,29 @@ Licence: Redistributable. See LICENCE.ueagle-atm4-firmware for details
Driver: ueagle-atm - Driver for USB ADSL Modems based on Eagle I,II,III
File: ueagle-atm/930-fpga.bin
-File: ueagle-atm/CMVeiWO.bin
-File: ueagle-atm/CMVepFR10.bin
-File: ueagle-atm/DSP9p.bin
+File: ueagle-atm/CMVeiWO.bin
+File: ueagle-atm/CMVepFR10.bin
+File: ueagle-atm/DSP9p.bin
File: ueagle-atm/eagleIII.fw
-File: ueagle-atm/adi930.fw
-File: ueagle-atm/CMVep.bin
-File: ueagle-atm/CMVepFR.bin
+File: ueagle-atm/adi930.fw
+File: ueagle-atm/CMVep.bin
+File: ueagle-atm/CMVepFR.bin
File: ueagle-atm/DSPei.bin
-File: ueagle-atm/CMV9i.bin
-File: ueagle-atm/CMVepES03.bin
-File: ueagle-atm/CMVepIT.bin
+File: ueagle-atm/CMV9i.bin
+File: ueagle-atm/CMVepES03.bin
+File: ueagle-atm/CMVepIT.bin
File: ueagle-atm/DSPep.bin
-File: ueagle-atm/CMV9p.bin
-File: ueagle-atm/CMVepES.bin
-File: ueagle-atm/CMVepWO.bin
+File: ueagle-atm/CMV9p.bin
+File: ueagle-atm/CMVepES.bin
+File: ueagle-atm/CMVepWO.bin
File: ueagle-atm/eagleI.fw
-File: ueagle-atm/CMVei.bin
-File: ueagle-atm/CMVepFR04.bin
-File: ueagle-atm/DSP9i.bin
+File: ueagle-atm/CMVei.bin
+File: ueagle-atm/CMVepFR04.bin
+File: ueagle-atm/DSP9i.bin
File: ueagle-atm/eagleII.fw
Version: 1.1
-Licence: Redistributable. Based on
+Licence: Redistributable. Based on
https://mail.gna.org/public/eagleusb-dev/2004-11/msg00172.html
--------------------------------------------------------------------------
@@ -3949,7 +3949,7 @@ License:
This SDD ("Static Dynamic Data") file is licensed strictly for use with
the Sagrad WiFi modules (such as the SG901-1091/1098) that utilize the
- cw1200 driver. There is no warranty expressed or implied about its
+ cw1200 driver. There is no warranty expressed or implied about its
fitness for any purpose.
Permission is hereby granted for the distribution of this SDD file as
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 02/15] WHENCE: remove unnecessary filename quotation
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
2023-02-18 18:46 ` [PATCH 01/15] WHENCE: remove trailing white space Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 03/15] check_whence, WHENCE, copy-firmware: escape filenames with spaces Emil Velikov
` (13 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
WHENCE | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
diff --git a/WHENCE b/WHENCE
index 23c8847..fe5e423 100644
--- a/WHENCE
+++ b/WHENCE
@@ -2691,20 +2691,20 @@ Link: brcm/brcmfmac54591-pcie.clm_blob -> ../cypress/cyfmac54591-pcie.clm_blob
Licence: Redistributable. See LICENCE.cypress for details.
File: "brcm/brcmfmac43241b4-sdio.Intel Corp.-VALLEYVIEW C0 PLATFORM.txt"
-File: "brcm/brcmfmac4330-sdio.Prowise-PT301.txt"
+File: brcm/brcmfmac4330-sdio.Prowise-PT301.txt
File: "brcm/brcmfmac43340-sdio.ASUSTeK COMPUTER INC.-TF103CE.txt"
-File: "brcm/brcmfmac43340-sdio.meegopad-t08.txt"
-File: "brcm/brcmfmac43340-sdio.pov-tab-p1006w-data.txt"
-File: "brcm/brcmfmac43340-sdio.predia-basic.txt"
-File: "brcm/brcmfmac43362-sdio.WC121.txt"
-File: "brcm/brcmfmac43362-sdio.cubietech,cubietruck.txt"
+File: brcm/brcmfmac43340-sdio.meegopad-t08.txt
+File: brcm/brcmfmac43340-sdio.pov-tab-p1006w-data.txt
+File: brcm/brcmfmac43340-sdio.predia-basic.txt
+File: brcm/brcmfmac43362-sdio.WC121.txt
+File: brcm/brcmfmac43362-sdio.cubietech,cubietruck.txt
Link: brcm/brcmfmac43362-sdio.kobo,aura.txt -> brcmfmac43362-sdio.WC121.txt
Link: brcm/brcmfmac43362-sdio.kobo,tolino-shine2hd.txt -> brcmfmac43362-sdio.WC121.txt
Link: brcm/brcmfmac43362-sdio.lemaker,bananapro.txt -> brcmfmac43362-sdio.cubietech,cubietruck.txt
-File: "brcm/brcmfmac43430a0-sdio.ilife-S806.txt"
-File: "brcm/brcmfmac43430a0-sdio.jumper-ezpad-mini3.txt"
+File: brcm/brcmfmac43430a0-sdio.ilife-S806.txt
+File: brcm/brcmfmac43430a0-sdio.jumper-ezpad-mini3.txt
File: "brcm/brcmfmac43430a0-sdio.ONDA-V80 PLUS.txt"
-File: "brcm/brcmfmac43430-sdio.AP6212.txt"
+File: brcm/brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.sinovoip,bpi-m2-plus.txt -> brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.sinovoip,bpi-m2-zero.txt -> brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.sinovoip,bpi-m2-ultra.txt -> brcmfmac43430-sdio.AP6212.txt
@@ -2713,19 +2713,19 @@ Link: brcm/brcmfmac43430-sdio.friendlyarm,nanopi-r1.txt -> brcmfmac43430-sdio.AP
Link: brcm/brcmfmac43430-sdio.starfive,visionfive-v1.txt -> brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.beagle,beaglev-starlight-jh7100-a1.txt -> brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.beagle,beaglev-starlight-jh7100-r0.txt -> brcmfmac43430-sdio.AP6212.txt
-File: "brcm/brcmfmac43430-sdio.Hampoo-D2D3_Vi8A1.txt"
-File: "brcm/brcmfmac43430-sdio.MUR1DX.txt"
-File: "brcm/brcmfmac43430-sdio.raspberrypi,3-model-b.txt"
+File: brcm/brcmfmac43430-sdio.Hampoo-D2D3_Vi8A1.txt
+File: brcm/brcmfmac43430-sdio.MUR1DX.txt
+File: brcm/brcmfmac43430-sdio.raspberrypi,3-model-b.txt
Link: brcm/brcmfmac43430-sdio.raspberrypi,model-zero-w.txt -> brcmfmac43430-sdio.raspberrypi,3-model-b.txt
Link: brcm/brcmfmac43430-sdio.raspberrypi,model-zero-2-w.txt -> brcmfmac43430-sdio.raspberrypi,3-model-b.txt
-File: "brcm/brcmfmac43455-sdio.acepc-t8.txt"
-File: "brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt"
+File: brcm/brcmfmac43455-sdio.acepc-t8.txt
+File: brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt
Link: brcm/brcmfmac43455-sdio.raspberrypi,3-model-a-plus.txt -> brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt
-File: "brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt"
+File: brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
Link: brcm/brcmfmac43455-sdio.Raspberry\ Pi\ Foundation-Raspberry\ Pi\ 4\ Model\ B.txt -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt
Link: brcm/brcmfmac43455-sdio.Raspberry\ Pi\ Foundation-Raspberry\ Pi\ Compute\ Module\ 4.txt -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt
File: "brcm/brcmfmac43455-sdio.MINIX-NEO Z83-4.txt"
-File: "brcm/brcmfmac4356-pcie.gpd-win-pocket.txt"
+File: brcm/brcmfmac4356-pcie.gpd-win-pocket.txt
File: "brcm/brcmfmac4356-pcie.Xiaomi Inc-Mipad2.txt"
File: brcm/brcmfmac4356-sdio.AP6356S.txt
Link: brcm/brcmfmac4356-sdio.firefly,firefly-rk3399.txt -> brcmfmac4356-sdio.AP6356S.txt
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 03/15] check_whence, WHENCE, copy-firmware: escape filenames with spaces
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
2023-02-18 18:46 ` [PATCH 01/15] WHENCE: remove trailing white space Emil Velikov
2023-02-18 18:46 ` [PATCH 02/15] WHENCE: remove unnecessary filename quotation Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 04/15] WHENCE: remove duplicate File entries Emil Velikov
` (12 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
We have a handful of instances which have space in the filename - just
escape them (like we do for the symlinks) and drop the extra re pattern.
With this we can also drop the extra sed pattern, stripping out the
quotation marks.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
WHENCE | 10 +++++-----
check_whence.py | 8 ++------
copy-firmware.sh | 2 +-
3 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/WHENCE b/WHENCE
index fe5e423..13013f8 100644
--- a/WHENCE
+++ b/WHENCE
@@ -2690,9 +2690,9 @@ Link: brcm/brcmfmac54591-pcie.clm_blob -> ../cypress/cyfmac54591-pcie.clm_blob
Licence: Redistributable. See LICENCE.cypress for details.
-File: "brcm/brcmfmac43241b4-sdio.Intel Corp.-VALLEYVIEW C0 PLATFORM.txt"
+File: brcm/brcmfmac43241b4-sdio.Intel\ Corp.-VALLEYVIEW\ C0\ PLATFORM.txt
File: brcm/brcmfmac4330-sdio.Prowise-PT301.txt
-File: "brcm/brcmfmac43340-sdio.ASUSTeK COMPUTER INC.-TF103CE.txt"
+File: brcm/brcmfmac43340-sdio.ASUSTeK\ COMPUTER\ INC.-TF103CE.txt
File: brcm/brcmfmac43340-sdio.meegopad-t08.txt
File: brcm/brcmfmac43340-sdio.pov-tab-p1006w-data.txt
File: brcm/brcmfmac43340-sdio.predia-basic.txt
@@ -2703,7 +2703,7 @@ Link: brcm/brcmfmac43362-sdio.kobo,tolino-shine2hd.txt -> brcmfmac43362-sdio.WC1
Link: brcm/brcmfmac43362-sdio.lemaker,bananapro.txt -> brcmfmac43362-sdio.cubietech,cubietruck.txt
File: brcm/brcmfmac43430a0-sdio.ilife-S806.txt
File: brcm/brcmfmac43430a0-sdio.jumper-ezpad-mini3.txt
-File: "brcm/brcmfmac43430a0-sdio.ONDA-V80 PLUS.txt"
+File: brcm/brcmfmac43430a0-sdio.ONDA-V80\ PLUS.txt
File: brcm/brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.sinovoip,bpi-m2-plus.txt -> brcmfmac43430-sdio.AP6212.txt
Link: brcm/brcmfmac43430-sdio.sinovoip,bpi-m2-zero.txt -> brcmfmac43430-sdio.AP6212.txt
@@ -2724,9 +2724,9 @@ Link: brcm/brcmfmac43455-sdio.raspberrypi,3-model-a-plus.txt -> brcmfmac43455-sd
File: brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
Link: brcm/brcmfmac43455-sdio.Raspberry\ Pi\ Foundation-Raspberry\ Pi\ 4\ Model\ B.txt -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt
Link: brcm/brcmfmac43455-sdio.Raspberry\ Pi\ Foundation-Raspberry\ Pi\ Compute\ Module\ 4.txt -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt
-File: "brcm/brcmfmac43455-sdio.MINIX-NEO Z83-4.txt"
+File: brcm/brcmfmac43455-sdio.MINIX-NEO\ Z83-4.txt
File: brcm/brcmfmac4356-pcie.gpd-win-pocket.txt
-File: "brcm/brcmfmac4356-pcie.Xiaomi Inc-Mipad2.txt"
+File: brcm/brcmfmac4356-pcie.Xiaomi\ Inc-Mipad2.txt
File: brcm/brcmfmac4356-sdio.AP6356S.txt
Link: brcm/brcmfmac4356-sdio.firefly,firefly-rk3399.txt -> brcmfmac4356-sdio.AP6356S.txt
Link: brcm/brcmfmac4356-sdio.khadas,vim2.txt -> brcmfmac4356-sdio.AP6356S.txt
diff --git a/check_whence.py b/check_whence.py
index 8805e99..f347f0e 100755
--- a/check_whence.py
+++ b/check_whence.py
@@ -6,13 +6,9 @@ from io import open
def list_whence():
with open('WHENCE', encoding='utf-8') as whence:
for line in whence:
- match = re.match(r'(?:File|Source):\s*"(.*)"', line)
+ match = re.match(r'(?:File|Source):\s*(.*)', line)
if match:
- yield match.group(1)
- continue
- match = re.match(r'(?:File|Source):\s*(\S*)', line)
- if match:
- yield match.group(1)
+ yield match.group(1).replace("\ ", " ")
continue
match = re.match(r'Licen[cs]e: (?:.*\bSee (.*) for details\.?|(\S*))\n',
line)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index bbacb92..6bceef3 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -31,7 +31,7 @@ while test $# -gt 0; do
esac
done
-grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
+grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
test -f "$f" || continue
$verbose "copying file $f"
install -d $destdir/$(dirname "$f")
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 04/15] WHENCE: remove duplicate File entries
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (2 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 03/15] check_whence, WHENCE, copy-firmware: escape filenames with spaces Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 05/15] check_whence: error on duplicate file entries Emil Velikov
` (11 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
XXX: still one outstanding - mediatek/mt7663pr2h.bin
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
WHENCE | 19 -------------------
1 file changed, 19 deletions(-)
diff --git a/WHENCE b/WHENCE
index 13013f8..d1cc3f8 100644
--- a/WHENCE
+++ b/WHENCE
@@ -1073,10 +1073,6 @@ Version: 75.27f1c37b.0
File: iwlwifi-so-a0-jf-b0-77.ucode
Version: 74.f92b5fed.0
-File: iwlwifi-so-a0-gf-a0.pnvm
-
-File: iwlwifi-so-a0-gf4-a0.pnvm
-
File: iwlwifi-ty-a0-gf-a0.pnvm
Licence: Redistributable. See LICENCE.iwlwifi_firmware for details
@@ -2980,19 +2976,6 @@ Licence: Redistributable. See LICENCE.rtlwifi_firmware.txt for details.
--------------------------------------------------------------------------
-Driver: r8723au - Realtek 802.11n WLAN driver for RTL8723AU
-
-Info: Taken from Realtek driver rtl8723A_WiFi_linux_v4.1.3_6044.20121224
- Firmware is embedded in the driver as data statements. This info
- has been extracted into a binary file.
-File: rtlwifi/rtl8723aufw_A.bin
-File: rtlwifi/rtl8723aufw_B.bin
-File: rtlwifi/rtl8723aufw_B_NoBT.bin
-
-Licence: Redistributable. See LICENCE.rtlwifi_firmware.txt for details.
-
---------------------------------------------------------------------------
-
Driver: rtl8188ee - Realtek 802.11n WLAN driver for RTL8188EE
Info: Taken from Realtek version rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0010.0109.2013
@@ -5943,8 +5926,6 @@ Link: cirrus/cs35l41-dsp1-spk-prot-103c8995.wmfw -> cs35l41/v6.61.1/halo_cspl_RA
Link: cirrus/cs35l41-dsp1-spk-cali-103c8995.wmfw -> cs35l41/v6.61.1/halo_cspl_RAM_revB2_29.63.1.wmfw
Link: cirrus/cs35l41-dsp1-spk-prot-103c89c6.wmfw -> cs35l41/v6.61.1/halo_cspl_RAM_revB2_29.63.1.wmfw
Link: cirrus/cs35l41-dsp1-spk-cali-103c89c6.wmfw -> cs35l41/v6.61.1/halo_cspl_RAM_revB2_29.63.1.wmfw
-File: cirrus/cs35l41-dsp1-spk-cali-103c8971.bin
-File: cirrus/cs35l41-dsp1-spk-prot-103c8971.bin
File: cirrus/cs35l41-dsp1-spk-prot-103c8971.bin
File: cirrus/cs35l41-dsp1-spk-cali-103c8971.bin
File: cirrus/cs35l41-dsp1-spk-prot-103c8972.bin
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 05/15] check_whence: error on duplicate file entries
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (3 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 04/15] WHENCE: remove duplicate File entries Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 06/15] check_whence: error on directory listed as File Emil Velikov
` (10 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
There's little point in copying (or compressing with later patches) the
same files multiple times. So let's error out when duplicate entries are
present.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
check_whence.py | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/check_whence.py b/check_whence.py
index f347f0e..7ff21f6 100755
--- a/check_whence.py
+++ b/check_whence.py
@@ -24,6 +24,14 @@ def list_whence():
yield match.group(2)
continue
+def list_whence_files():
+ with open('WHENCE', encoding='utf-8') as whence:
+ for line in whence:
+ match = re.match(r'File:\s*(.*)', line)
+ if match:
+ yield match.group(1).replace("\ ", " ")
+ continue
+
def list_git():
with os.popen('git ls-files') as git_files:
for line in git_files:
@@ -32,12 +40,17 @@ def list_git():
def main():
ret = 0
whence_list = list(list_whence())
+ whence_files = list(list_whence_files())
known_files = set(name for name in whence_list if not name.endswith('/')) | \
set(['check_whence.py', 'configure', 'Makefile',
'README', 'copy-firmware.sh', 'WHENCE'])
known_prefixes = set(name for name in whence_list if name.endswith('/'))
git_files = set(list_git())
+ for name in set(fw for fw in whence_files if whence_files.count(fw) > 1):
+ sys.stderr.write('E: %s listed in WHENCE twice\n' % name)
+ ret = 1
+
for name in sorted(list(known_files - git_files)):
sys.stderr.write('E: %s listed in WHENCE does not exist\n' % name)
ret = 1
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 06/15] check_whence: error on directory listed as File
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (4 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 05/15] check_whence: error on duplicate file entries Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 07/15] copy-firmware: remove non-applicable file presence test Emil Velikov
` (9 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
No occurrences in-tree, but it makes sense to sanity check that.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
check_whence.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/check_whence.py b/check_whence.py
index 7ff21f6..130966e 100755
--- a/check_whence.py
+++ b/check_whence.py
@@ -47,6 +47,11 @@ def main():
known_prefixes = set(name for name in whence_list if name.endswith('/'))
git_files = set(list_git())
+ for name in set(name for name in whence_files if name.endswith('/')):
+ sys.stderr.write('E: %s listed in WHENCE as File, but is directory\n' %
+ name)
+ ret = 1
+
for name in set(fw for fw in whence_files if whence_files.count(fw) > 1):
sys.stderr.write('E: %s listed in WHENCE twice\n' % name)
ret = 1
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 07/15] copy-firmware: remove non-applicable file presence test
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (5 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 06/15] check_whence: error on directory listed as File Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 08/15] check_whence: error if File: is actually a link Emil Velikov
` (8 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
The check_whence.py script already ensures that the files listed in
WHENCE are present on-disk. Thus the extra check isn't needed.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 6bceef3..656b0ec 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -32,7 +32,6 @@ while test $# -gt 0; do
done
grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
- test -f "$f" || continue
$verbose "copying file $f"
install -d $destdir/$(dirname "$f")
cp -d "$f" $destdir/"$f"
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 08/15] check_whence: error if File: is actually a link
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (6 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 07/15] copy-firmware: remove non-applicable file presence test Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 09/15] check_whence: error if symlinks are in-tree Emil Velikov
` (7 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
This allows us to remove the -d aka --no-dereference --preserve=links
from the copy command.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
check_whence.py | 5 +++++
copy-firmware.sh | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/check_whence.py b/check_whence.py
index 130966e..92b104e 100755
--- a/check_whence.py
+++ b/check_whence.py
@@ -56,6 +56,11 @@ def main():
sys.stderr.write('E: %s listed in WHENCE twice\n' % name)
ret = 1
+ for name in set(link for link in whence_files if os.path.islink(link)):
+ sys.stderr.write('E: %s listed in WHENCE as File, but is a symlink\n' %
+ name)
+ ret = 1
+
for name in sorted(list(known_files - git_files)):
sys.stderr.write('E: %s listed in WHENCE does not exist\n' % name)
ret = 1
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 656b0ec..e3f3176 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -34,7 +34,7 @@ done
grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
$verbose "copying file $f"
install -d $destdir/$(dirname "$f")
- cp -d "$f" $destdir/"$f"
+ cp "$f" $destdir/"$f"
done
grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 09/15] check_whence: error if symlinks are in-tree
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (7 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 08/15] check_whence: error if File: is actually a link Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 10/15] copy-firmware: remove unreachable symlink workarounds Emil Velikov
` (6 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Currently we have no symlinks in-tree. Add a simple check, ensuring they
don't get added in the future.
This allows us to remove the clunky symlink checking code in
copy-firmware.sh
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
check_whence.py | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/check_whence.py b/check_whence.py
index 92b104e..75fd3ad 100755
--- a/check_whence.py
+++ b/check_whence.py
@@ -32,6 +32,14 @@ def list_whence_files():
yield match.group(1).replace("\ ", " ")
continue
+def list_whence_links():
+ with open('WHENCE', encoding='utf-8') as whence:
+ for line in whence:
+ match = re.match(r'Link:\s*(.*) ->', line)
+ if match:
+ yield match.group(1).replace("\ ", " ")
+ continue
+
def list_git():
with os.popen('git ls-files') as git_files:
for line in git_files:
@@ -41,6 +49,7 @@ def main():
ret = 0
whence_list = list(list_whence())
whence_files = list(list_whence_files())
+ whence_links = list(list_whence_links())
known_files = set(name for name in whence_list if not name.endswith('/')) | \
set(['check_whence.py', 'configure', 'Makefile',
'README', 'copy-firmware.sh', 'WHENCE'])
@@ -61,6 +70,10 @@ def main():
name)
ret = 1
+ for name in set(link for link in whence_links if os.path.islink(link)):
+ sys.stderr.write('E: %s listed in WHENCE as Link, is in tree\n' % name)
+ ret = 1
+
for name in sorted(list(known_files - git_files)):
sys.stderr.write('E: %s listed in WHENCE does not exist\n' % name)
ret = 1
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 10/15] copy-firmware: remove unreachable symlink workarounds
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (8 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 09/15] check_whence: error if symlinks are in-tree Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 11/15] copy-firmware: quote deskdir to prevent word splitting Emil Velikov
` (5 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
There are no symlinks in-tree and the earlier commit to check_whence.py
ensures new ones don't get added. Thus we can remove the consistency
checks and pruning code.
With this done we can also drop:
- the --force argument for ln
- the prune argument and related handling
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 36 +++---------------------------------
1 file changed, 3 insertions(+), 33 deletions(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index e3f3176..0132244 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -5,7 +5,6 @@
#
verbose=:
-prune=no
while test $# -gt 0; do
case $1 in
@@ -14,11 +13,6 @@ while test $# -gt 0; do
shift
;;
- -P | --prune)
- prune=yes
- shift
- ;;
-
*)
if test "x$destdir" != "x"; then
echo "ERROR: unknown command-line options: $@"
@@ -38,33 +32,9 @@ grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
done
grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
- if test -L "$f"; then
- test -f "$destdir/$f" && continue
- $verbose "copying link $f"
- install -d $destdir/$(dirname "$f")
- cp -d "$f" $destdir/"$f"
-
- if test "x$d" != "x"; then
- target=`readlink "$f"`
-
- if test "x$target" != "x$d"; then
- $verbose "WARNING: inconsistent symlink target: $target != $d"
- else
- if test "x$prune" != "xyes"; then
- $verbose "WARNING: unneeded symlink detected: $f"
- else
- $verbose "WARNING: pruning unneeded symlink $f"
- rm -f "$f"
- fi
- fi
- else
- $verbose "WARNING: missing target for symlink $f"
- fi
- else
- $verbose "creating link $f -> $d"
- install -d $destdir/$(dirname "$f")
- ln -sf "$d" "$destdir/$f"
- fi
+ $verbose "creating link $f -> $d"
+ install -d $destdir/$(dirname "$f")
+ ln -s "$d" "$destdir/$f"
done
exit 0
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 11/15] copy-firmware: quote deskdir to prevent word splitting
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (9 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 10/15] copy-firmware: remove unreachable symlink workarounds Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 12/15] copy-firmware: tweak sed invocation Emil Velikov
` (4 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 0132244..c6a2fe9 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -27,13 +27,13 @@ done
grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
$verbose "copying file $f"
- install -d $destdir/$(dirname "$f")
- cp "$f" $destdir/"$f"
+ install -d "$destdir"/$(dirname "$f")
+ cp "$f" "$destdir/$f"
done
grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
$verbose "creating link $f -> $d"
- install -d $destdir/$(dirname "$f")
+ install -d "$destdir"/$(dirname "$f")
ln -s "$d" "$destdir/$f"
done
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 12/15] copy-firmware: tweak sed invocation
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (10 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 11/15] copy-firmware: quote deskdir to prevent word splitting Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 13/15] copy-firmware: quote the output of dirname Emil Velikov
` (3 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Add space between the arguments and pattern and combine patterns where
possible.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index c6a2fe9..722227f 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -25,13 +25,13 @@ while test $# -gt 0; do
esac
done
-grep '^File:' WHENCE | sed -e's/^File: *//g' | while read f; do
+grep '^File:' WHENCE | sed -e 's/^File: *//g' | while read f; do
$verbose "copying file $f"
install -d "$destdir"/$(dirname "$f")
cp "$f" "$destdir/$f"
done
-grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
+grep -E '^Link:' WHENCE | sed -e 's/^Link: *//g;s/-> //g' | while read f d; do
$verbose "creating link $f -> $d"
install -d "$destdir"/$(dirname "$f")
ln -s "$d" "$destdir/$f"
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 13/15] copy-firmware: quote the output of dirname
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (11 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 12/15] copy-firmware: tweak sed invocation Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 14/15] copy-firmware: silence the last shellcheck warnings Emil Velikov
` (2 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
As highlighted by shellcheck - as-is one could get word splitting. That
isn't an issue, yet, since none of the in-tree directories have space in
their name.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 722227f..5a5e1a8 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -27,13 +27,13 @@ done
grep '^File:' WHENCE | sed -e 's/^File: *//g' | while read f; do
$verbose "copying file $f"
- install -d "$destdir"/$(dirname "$f")
+ install -d "$destdir/$(dirname "$f")"
cp "$f" "$destdir/$f"
done
grep -E '^Link:' WHENCE | sed -e 's/^Link: *//g;s/-> //g' | while read f d; do
$verbose "creating link $f -> $d"
- install -d "$destdir"/$(dirname "$f")
+ install -d "$destdir/$(dirname "$f")"
ln -s "$d" "$destdir/$f"
done
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 14/15] copy-firmware: silence the last shellcheck warnings
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (12 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 13/15] copy-firmware: quote the output of dirname Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-18 18:46 ` [PATCH 15/15] Makefile, copy-firmware: support xz/zstd compressed firmware Emil Velikov
2023-02-28 14:18 ` [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
copy-firmware.sh | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 5a5e1a8..a52b847 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -9,13 +9,14 @@ verbose=:
while test $# -gt 0; do
case $1 in
-v | --verbose)
+ # shellcheck disable=SC2209
verbose=echo
shift
;;
*)
if test "x$destdir" != "x"; then
- echo "ERROR: unknown command-line options: $@"
+ echo "ERROR: unknown command-line options: $*"
exit 1
fi
@@ -25,12 +26,14 @@ while test $# -gt 0; do
esac
done
+# shellcheck disable=SC2162
grep '^File:' WHENCE | sed -e 's/^File: *//g' | while read f; do
$verbose "copying file $f"
install -d "$destdir/$(dirname "$f")"
cp "$f" "$destdir/$f"
done
+# shellcheck disable=SC2162
grep -E '^Link:' WHENCE | sed -e 's/^Link: *//g;s/-> //g' | while read f d; do
$verbose "creating link $f -> $d"
install -d "$destdir/$(dirname "$f")"
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 15/15] Makefile, copy-firmware: support xz/zstd compressed firmware
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (13 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 14/15] copy-firmware: silence the last shellcheck warnings Emil Velikov
@ 2023-02-18 18:46 ` Emil Velikov
2023-02-28 14:18 ` [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
15 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-18 18:46 UTC (permalink / raw)
To: linux-firmware; +Cc: emil.l.velikov
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
---
Makefile | 8 ++++++++
copy-firmware.sh | 33 +++++++++++++++++++++++++++++----
2 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile
index 83a0ec6..50a4c29 100644
--- a/Makefile
+++ b/Makefile
@@ -11,3 +11,11 @@ check:
install:
install -d $(DESTDIR)$(FIRMWAREDIR)
./copy-firmware.sh $(DESTDIR)$(FIRMWAREDIR)
+
+install-xz:
+ install -d $(DESTDIR)$(FIRMWAREDIR)
+ ./copy-firmware.sh --xz $(DESTDIR)$(FIRMWAREDIR)
+
+install-zst:
+ install -d $(DESTDIR)$(FIRMWAREDIR)
+ ./copy-firmware.sh --zstd $(DESTDIR)$(FIRMWAREDIR)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index a52b847..6c78e27 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -5,6 +5,9 @@
#
verbose=:
+# shellcheck disable=SC2209
+compress=cat
+compext=
while test $# -gt 0; do
case $1 in
@@ -14,6 +17,28 @@ while test $# -gt 0; do
shift
;;
+ --xz)
+ if test "x$compext" == "x.zst"; then
+ echo "ERROR: cannot mix XZ and ZSTD compression"
+ exit 1
+ fi
+ # shellcheck disable=SC2209
+ compress="xz --compress --quiet --stdout --check=crc32"
+ compext=".xz"
+ shift
+ ;;
+
+ --zstd)
+ if test "x$compext" == "x.xz"; then
+ echo "ERROR: cannot mix XZ and ZSTD compression"
+ exit 1
+ fi
+ # shellcheck disable=SC2209
+ compress="zstd --compress --quiet --stdout"
+ compext=".zst"
+ shift
+ ;;
+
*)
if test "x$destdir" != "x"; then
echo "ERROR: unknown command-line options: $*"
@@ -28,16 +53,16 @@ done
# shellcheck disable=SC2162
grep '^File:' WHENCE | sed -e 's/^File: *//g' | while read f; do
- $verbose "copying file $f"
install -d "$destdir/$(dirname "$f")"
- cp "$f" "$destdir/$f"
+ $verbose "copying/compressing file $f$compext"
+ $compress "$f" > "$destdir/$f$compext"
done
# shellcheck disable=SC2162
grep -E '^Link:' WHENCE | sed -e 's/^Link: *//g;s/-> //g' | while read f d; do
- $verbose "creating link $f -> $d"
install -d "$destdir/$(dirname "$f")"
- ln -s "$d" "$destdir/$f"
+ $verbose "creating link $f$compext -> $d$compext"
+ ln -s "$d$compext" "$destdir/$f$compext"
done
exit 0
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
` (14 preceding siblings ...)
2023-02-18 18:46 ` [PATCH 15/15] Makefile, copy-firmware: support xz/zstd compressed firmware Emil Velikov
@ 2023-02-28 14:18 ` Emil Velikov
2023-02-28 14:36 ` David Woodhouse
15 siblings, 1 reply; 23+ messages in thread
From: Emil Velikov @ 2023-02-28 14:18 UTC (permalink / raw)
To: linux-firmware, jwboyer
On Sat, 18 Feb 2023 at 18:46, Emil Velikov <emil.l.velikov@gmail.com> wrote:
>
> Greetings everyone,
>
> For a while now, the kernel has been accepting compressed firmware - be that xz
> or zstd. There is a patch for the former on the ML, yet the latter is still
> lacking.
>
> While working through I've noticed a few issues - of various severity - so
> decided to fix them all.
>
> Mind you, I'm sending everything as single patch series to gauge feedback. I'm
> quite happy to split and rework as per the community's feedback.
>
> In a gist:
> - remove quoted filenames in WHENCE - escape white space as needed
> - remove _nearly_ all duplicate entries in WHENCE && add check
> - NOTE: one entry remains, which I'm not sure how to resolve cleanly
> - check if directory is listed as File - had a mistake like that locally
> - check we have no symlinks and drop symlink workarounds
> - quote deskdir (genuine bug) + few more shellcheck inspired fixes
> - support xz and zstd compressed firmware - no intermediate files, pipe
> directly into destdir
>
>
> Kind regards
> Emil
>
>
> Emil Velikov (15):
> WHENCE: remove trailing white space
> WHENCE: remove unnecessary filename quotation
> check_whence, WHENCE, copy-firmware: escape filenames with spaces
> WHENCE: remove duplicate File entries
> check_whence: error on duplicate file entries
> check_whence: error on directory listed as File
> copy-firmware: remove non-applicable file presence test
> check_whence: error if File: is actually a link
> check_whence: error if symlinks are in-tree
> copy-firmware: remove unreachable symlink workarounds
> copy-firmware: quote deskdir to prevent word splitting
> copy-firmware: tweak sed invocation
> copy-firmware: quote the output of dirname
> copy-firmware: silence the last shellcheck warnings
> Makefile, copy-firmware: support xz/zstd compressed firmware
>
> Makefile | 8 ++++
> WHENCE | 97 +++++++++++++++++++-----------------------------
> check_whence.py | 44 +++++++++++++++++++---
> copy-firmware.sh | 71 +++++++++++++++++------------------
> 4 files changed, 119 insertions(+), 101 deletions(-)
>
Adding Josh Boyer - the maintainer, according to git log.
Josh, do you have any objections to the patches - part or in full? Can
you pick these off the ML or shall I send you a git request-pull?
Thanks
Emil
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 14:18 ` [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
@ 2023-02-28 14:36 ` David Woodhouse
2023-02-28 16:46 ` Emil Velikov
0 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2023-02-28 14:36 UTC (permalink / raw)
To: Emil Velikov, linux-firmware, jwboyer
[-- Attachment #1: Type: text/plain, Size: 2718 bytes --]
On Tue, 2023-02-28 at 14:18 +0000, Emil Velikov wrote:
> On Sat, 18 Feb 2023 at 18:46, Emil Velikov <emil.l.velikov@gmail.com>
> wrote:
> >
> > Greetings everyone,
> >
> > For a while now, the kernel has been accepting compressed firmware - be that xz
> > or zstd. There is a patch for the former on the ML, yet the latter is still
> > lacking.
> >
> > While working through I've noticed a few issues - of various severity - so
> > decided to fix them all.
> >
> > Mind you, I'm sending everything as single patch series to gauge feedback. I'm
> > quite happy to split and rework as per the community's feedback.
> >
> > In a gist:
> > - remove quoted filenames in WHENCE - escape white space as needed
> > - remove _nearly_ all duplicate entries in WHENCE && add check
> > - NOTE: one entry remains, which I'm not sure how to resolve cleanly
> > - check if directory is listed as File - had a mistake like that locally
> > - check we have no symlinks and drop symlink workarounds
> > - quote deskdir (genuine bug) + few more shellcheck inspired fixes
> > - support xz and zstd compressed firmware - no intermediate files, pipe
> > directly into destdir
> >
> >
> > Kind regards
> > Emil
> >
> >
> > Emil Velikov (15):
> > WHENCE: remove trailing white space
> > WHENCE: remove unnecessary filename quotation
> > check_whence, WHENCE, copy-firmware: escape filenames with spaces
> > WHENCE: remove duplicate File entries
> > check_whence: error on duplicate file entries
> > check_whence: error on directory listed as File
> > copy-firmware: remove non-applicable file presence test
> > check_whence: error if File: is actually a link
> > check_whence: error if symlinks are in-tree
> > copy-firmware: remove unreachable symlink workarounds
> > copy-firmware: quote deskdir to prevent word splitting
> > copy-firmware: tweak sed invocation
> > copy-firmware: quote the output of dirname
> > copy-firmware: silence the last shellcheck warnings
> > Makefile, copy-firmware: support xz/zstd compressed firmware
> >
> > Makefile | 8 ++++
> > WHENCE | 97 +++++++++++++++++++-----------------------------
> > check_whence.py | 44 +++++++++++++++++++---
> > copy-firmware.sh | 71 +++++++++++++++++------------------
> > 4 files changed, 119 insertions(+), 101 deletions(-)
> >
>
> Adding Josh Boyer - the maintainer, according to git log.
>
> Josh, do you have any objections to the patches - part or in full? Can
> you pick these off the ML or shall I send you a git request-pull?
Does this work even for early microcode loading?
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 14:36 ` David Woodhouse
@ 2023-02-28 16:46 ` Emil Velikov
2023-02-28 16:48 ` David Woodhouse
0 siblings, 1 reply; 23+ messages in thread
From: Emil Velikov @ 2023-02-28 16:46 UTC (permalink / raw)
To: David Woodhouse; +Cc: linux-firmware, jwboyer
On Tue, 28 Feb 2023 at 14:37, David Woodhouse <dwmw2@infradead.org> wrote:
>
> On Tue, 2023-02-28 at 14:18 +0000, Emil Velikov wrote:
> > On Sat, 18 Feb 2023 at 18:46, Emil Velikov <emil.l.velikov@gmail.com>
> > wrote:
> > >
> > > Greetings everyone,
> > >
> > > For a while now, the kernel has been accepting compressed firmware - be that xz
> > > or zstd. There is a patch for the former on the ML, yet the latter is still
> > > lacking.
> > >
> > > While working through I've noticed a few issues - of various severity - so
> > > decided to fix them all.
> > >
> > > Mind you, I'm sending everything as single patch series to gauge feedback. I'm
> > > quite happy to split and rework as per the community's feedback.
> > >
> > > In a gist:
> > > - remove quoted filenames in WHENCE - escape white space as needed
> > > - remove _nearly_ all duplicate entries in WHENCE && add check
> > > - NOTE: one entry remains, which I'm not sure how to resolve cleanly
> > > - check if directory is listed as File - had a mistake like that locally
> > > - check we have no symlinks and drop symlink workarounds
> > > - quote deskdir (genuine bug) + few more shellcheck inspired fixes
> > > - support xz and zstd compressed firmware - no intermediate files, pipe
> > > directly into destdir
> > >
> > >
> > > Kind regards
> > > Emil
> > >
> > >
> > > Emil Velikov (15):
> > > WHENCE: remove trailing white space
> > > WHENCE: remove unnecessary filename quotation
> > > check_whence, WHENCE, copy-firmware: escape filenames with spaces
> > > WHENCE: remove duplicate File entries
> > > check_whence: error on duplicate file entries
> > > check_whence: error on directory listed as File
> > > copy-firmware: remove non-applicable file presence test
> > > check_whence: error if File: is actually a link
> > > check_whence: error if symlinks are in-tree
> > > copy-firmware: remove unreachable symlink workarounds
> > > copy-firmware: quote deskdir to prevent word splitting
> > > copy-firmware: tweak sed invocation
> > > copy-firmware: quote the output of dirname
> > > copy-firmware: silence the last shellcheck warnings
> > > Makefile, copy-firmware: support xz/zstd compressed firmware
> > >
> > > Makefile | 8 ++++
> > > WHENCE | 97 +++++++++++++++++++-----------------------------
> > > check_whence.py | 44 +++++++++++++++++++---
> > > copy-firmware.sh | 71 +++++++++++++++++------------------
> > > 4 files changed, 119 insertions(+), 101 deletions(-)
> > >
> >
> > Adding Josh Boyer - the maintainer, according to git log.
> >
> > Josh, do you have any objections to the patches - part or in full? Can
> > you pick these off the ML or shall I send you a git request-pull?
>
> Does this work even for early microcode loading?
>
Greetings David
Don't think I follow - the series is completely unrelated to microcode
loading. There isn't any mention of microcode in there, so I'm not
sure what inspired the question. Perhaps it was meant for another
thread?
Thanks
Emil
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 16:46 ` Emil Velikov
@ 2023-02-28 16:48 ` David Woodhouse
2023-02-28 17:02 ` Emil Velikov
0 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2023-02-28 16:48 UTC (permalink / raw)
To: Emil Velikov; +Cc: linux-firmware, jwboyer
[-- Attachment #1: Type: text/plain, Size: 366 bytes --]
On Tue, 2023-02-28 at 16:46 +0000, Emil Velikov wrote:
>
> Greetings David
>
> Don't think I follow - the series is completely unrelated to microcode
> loading. There isn't any mention of microcode in there, so I'm not
> sure what inspired the question. Perhaps it was meant for another
> thread?
cf. https://bugzilla.redhat.com/show_bug.cgi?id=2122271
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 16:48 ` David Woodhouse
@ 2023-02-28 17:02 ` Emil Velikov
2023-02-28 17:03 ` David Woodhouse
0 siblings, 1 reply; 23+ messages in thread
From: Emil Velikov @ 2023-02-28 17:02 UTC (permalink / raw)
To: David Woodhouse; +Cc: linux-firmware, jwboyer
On Tue, 28 Feb 2023 at 16:48, David Woodhouse <dwmw2@infradead.org> wrote:
>
> On Tue, 2023-02-28 at 16:46 +0000, Emil Velikov wrote:
> >
> > Greetings David
> >
> > Don't think I follow - the series is completely unrelated to microcode
> > loading. There isn't any mention of microcode in there, so I'm not
> > sure what inspired the question. Perhaps it was meant for another
> > thread?
>
> cf. https://bugzilla.redhat.com/show_bug.cgi?id=2122271
Nor really a RH/Fedora person, so the link is appreciated.
On my Arch machines, only the AMD PSP firmware is included in the
(mkinitcpio) generated init ramdisk. And as outlined in the bug that
works just fine. Will have a look into this.
Mind you - nearly all the patches can be applied independently of one
another. So dropping the final patch is one option in the short term.
Thanks again,
Emil
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 17:02 ` Emil Velikov
@ 2023-02-28 17:03 ` David Woodhouse
2023-02-28 17:20 ` Emil Velikov
0 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2023-02-28 17:03 UTC (permalink / raw)
To: Emil Velikov; +Cc: linux-firmware, jwboyer
[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]
On Tue, 2023-02-28 at 17:02 +0000, Emil Velikov wrote:
> On Tue, 28 Feb 2023 at 16:48, David Woodhouse <dwmw2@infradead.org> wrote:
> >
> > On Tue, 2023-02-28 at 16:46 +0000, Emil Velikov wrote:
> > >
> > > Greetings David
> > >
> > > Don't think I follow - the series is completely unrelated to microcode
> > > loading. There isn't any mention of microcode in there, so I'm not
> > > sure what inspired the question. Perhaps it was meant for another
> > > thread?
> >
> > cf. https://bugzilla.redhat.com/show_bug.cgi?id=2122271
>
> Nor really a RH/Fedora person, so the link is appreciated.
>
> On my Arch machines, only the AMD PSP firmware is included in the
> (mkinitcpio) generated init ramdisk. And as outlined in the bug that
> works just fine. Will have a look into this.
> Mind you - nearly all the patches can be applied independently of one
> another. So dropping the final patch is one option in the short term.
Maybe the answer is just to have a way to exempt *certain* files from
the compression?
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression
2023-02-28 17:03 ` David Woodhouse
@ 2023-02-28 17:20 ` Emil Velikov
0 siblings, 0 replies; 23+ messages in thread
From: Emil Velikov @ 2023-02-28 17:20 UTC (permalink / raw)
To: David Woodhouse; +Cc: linux-firmware, jwboyer, Adam Sampson
On Tue, 28 Feb 2023 at 17:03, David Woodhouse <dwmw2@infradead.org> wrote:
>
> On Tue, 2023-02-28 at 17:02 +0000, Emil Velikov wrote:
> > On Tue, 28 Feb 2023 at 16:48, David Woodhouse <dwmw2@infradead.org> wrote:
> > >
> > > On Tue, 2023-02-28 at 16:46 +0000, Emil Velikov wrote:
> > > >
> > > > Greetings David
> > > >
> > > > Don't think I follow - the series is completely unrelated to microcode
> > > > loading. There isn't any mention of microcode in there, so I'm not
> > > > sure what inspired the question. Perhaps it was meant for another
> > > > thread?
> > >
> > > cf. https://bugzilla.redhat.com/show_bug.cgi?id=2122271
> >
> > Nor really a RH/Fedora person, so the link is appreciated.
> >
> > On my Arch machines, only the AMD PSP firmware is included in the
> > (mkinitcpio) generated init ramdisk. And as outlined in the bug that
> > works just fine. Will have a look into this.
> > Mind you - nearly all the patches can be applied independently of one
> > another. So dropping the final patch is one option in the short term.
>
> Maybe the answer is just to have a way to exempt *certain* files from
> the compression?
Arch has been concatenating the amd ucode into a standalone initrd for
5+ years, thus making sure dracut or similar tools don't interfere.
Having some way to opt-out, would overall be beneficial. This way one
doesn't depend on distro specific solutions/hacks.
Skimming through the list - there's another set of WHENCE related
fixes [1], with a slightly better helper. Let me re-spin my series, so
things are less likely to clash.
-Emil
[1] https://lore.kernel.org/linux-firmware/20230216005602.18838-1-ats@offog.org/T/#t
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2023-02-28 17:20 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-18 18:46 [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
2023-02-18 18:46 ` [PATCH 01/15] WHENCE: remove trailing white space Emil Velikov
2023-02-18 18:46 ` [PATCH 02/15] WHENCE: remove unnecessary filename quotation Emil Velikov
2023-02-18 18:46 ` [PATCH 03/15] check_whence, WHENCE, copy-firmware: escape filenames with spaces Emil Velikov
2023-02-18 18:46 ` [PATCH 04/15] WHENCE: remove duplicate File entries Emil Velikov
2023-02-18 18:46 ` [PATCH 05/15] check_whence: error on duplicate file entries Emil Velikov
2023-02-18 18:46 ` [PATCH 06/15] check_whence: error on directory listed as File Emil Velikov
2023-02-18 18:46 ` [PATCH 07/15] copy-firmware: remove non-applicable file presence test Emil Velikov
2023-02-18 18:46 ` [PATCH 08/15] check_whence: error if File: is actually a link Emil Velikov
2023-02-18 18:46 ` [PATCH 09/15] check_whence: error if symlinks are in-tree Emil Velikov
2023-02-18 18:46 ` [PATCH 10/15] copy-firmware: remove unreachable symlink workarounds Emil Velikov
2023-02-18 18:46 ` [PATCH 11/15] copy-firmware: quote deskdir to prevent word splitting Emil Velikov
2023-02-18 18:46 ` [PATCH 12/15] copy-firmware: tweak sed invocation Emil Velikov
2023-02-18 18:46 ` [PATCH 13/15] copy-firmware: quote the output of dirname Emil Velikov
2023-02-18 18:46 ` [PATCH 14/15] copy-firmware: silence the last shellcheck warnings Emil Velikov
2023-02-18 18:46 ` [PATCH 15/15] Makefile, copy-firmware: support xz/zstd compressed firmware Emil Velikov
2023-02-28 14:18 ` [PATCH 00/15] Misc fixes, sanity checks and xz/zstd compression Emil Velikov
2023-02-28 14:36 ` David Woodhouse
2023-02-28 16:46 ` Emil Velikov
2023-02-28 16:48 ` David Woodhouse
2023-02-28 17:02 ` Emil Velikov
2023-02-28 17:03 ` David Woodhouse
2023-02-28 17:20 ` Emil Velikov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).