* [PATCH v2] Add support for compressing firmware in copy-firmware.sh
@ 2021-01-23 10:26 Peter Robinson
2021-02-08 13:47 ` Josh Boyer
0 siblings, 1 reply; 3+ messages in thread
From: Peter Robinson @ 2021-01-23 10:26 UTC (permalink / raw)
To: linux-firmware; +Cc: jwboyer, Peter Robinson
As of kernel 5.3 there's initial support for loading compressed firmware.
At this stage the only supported compression methis is "xz -C crc32" but
this option brings significant benefits.
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
---
v2: quote filename for xz command
Makefile | 4 ++++
copy-firmware.sh | 47 +++++++++++++++++++++++++++++++----------------
2 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/Makefile b/Makefile
index e1c362f..9a48471 100644
--- a/Makefile
+++ b/Makefile
@@ -11,3 +11,7 @@ check:
install:
mkdir -p $(DESTDIR)$(FIRMWAREDIR)
./copy-firmware.sh $(DESTDIR)$(FIRMWAREDIR)
+
+installcompress:
+ mkdir -p $(DESTDIR)$(FIRMWAREDIR)
+ ./copy-firmware.sh -C $(DESTDIR)$(FIRMWAREDIR)
diff --git a/copy-firmware.sh b/copy-firmware.sh
index 9b46b63..0dd2e5c 100755
--- a/copy-firmware.sh
+++ b/copy-firmware.sh
@@ -6,6 +6,7 @@
verbose=:
prune=no
+compress=no
while test $# -gt 0; do
case $1 in
@@ -19,6 +20,11 @@ while test $# -gt 0; do
shift
;;
+ -C | --compress)
+ compress=yes
+ shift
+ ;;
+
*)
if test "x$destdir" != "x"; then
echo "ERROR: unknown command-line options: $@"
@@ -31,40 +37,49 @@ while test $# -gt 0; do
esac
done
+if test "x$compress" = "xyes"; then
+ cmpxtn=.xz
+ grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
+ test -f "$f" || continue
+ $verbose "compressing $f"
+ xz -C crc32 "$f"
+ done
+fi
+
grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
- test -f "$f" || continue
- $verbose "copying file $f"
- mkdir -p $destdir/$(dirname "$f")
- cp -d "$f" $destdir/"$f"
+ test -f "$f$cmpxtn" || continue
+ $verbose "copying file $f$cmpxtn"
+ mkdir -p $destdir/$(dirname "$f$cmpxtn")
+ cp -d "$f$cmpxtn" $destdir/"$f$cmpxtn"
done
grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
- if test -L "$f"; then
- test -f "$destdir/$f" && continue
- $verbose "copying link $f"
- mkdir -p $destdir/$(dirname "$f")
+ if test -L "$f$cmpxtn"; then
+ test -f "$destdir/$f$cmpxtn" && continue
+ $verbose "copying link $f$cmpxtn"
+ mkdir -p $destdir/$(dirname "$f$cmpxtn")
cp -d "$f" $destdir/"$f"
if test "x$d" != "x"; then
- target=`readlink "$f"`
+ target=`readlink "$f$cmpxtn"`
if test "x$target" != "x$d"; then
$verbose "WARNING: inconsistent symlink target: $target != $d"
else
if test "x$prune" != "xyes"; then
- $verbose "WARNING: unneeded symlink detected: $f"
+ $verbose "WARNING: unneeded symlink detected: $f$cmpxtn"
else
- $verbose "WARNING: pruning unneeded symlink $f"
- rm -f "$f"
+ $verbose "WARNING: pruning unneeded symlink $f$cmpxtn"
+ rm -f "$f$cmpxtn"
fi
fi
else
- $verbose "WARNING: missing target for symlink $f"
+ $verbose "WARNING: missing target for symlink $f$cmpxtn"
fi
else
- $verbose "creating link $f -> $d"
- mkdir -p $destdir/$(dirname "$f")
- ln -sf "$d" "$destdir/$f"
+ $verbose "creating link $f$cmpxtn -> $d$cmpxtn"
+ mkdir -p $destdir/$(dirname "$f$cmpxtn")
+ ln -sf "$d$cmpxtn" "$destdir/$f$cmpxtn"
fi
done
--
2.29.2
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH v2] Add support for compressing firmware in copy-firmware.sh
2021-01-23 10:26 [PATCH v2] Add support for compressing firmware in copy-firmware.sh Peter Robinson
@ 2021-02-08 13:47 ` Josh Boyer
2021-02-08 13:50 ` Peter Robinson
0 siblings, 1 reply; 3+ messages in thread
From: Josh Boyer @ 2021-02-08 13:47 UTC (permalink / raw)
To: Peter Robinson; +Cc: Linux Firmware
On Sat, Jan 23, 2021 at 5:26 AM Peter Robinson <pbrobinson@gmail.com> wrote:
>
> As of kernel 5.3 there's initial support for loading compressed firmware.
> At this stage the only supported compression methis is "xz -C crc32" but
> this option brings significant benefits.
>
> Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
> ---
>
> v2: quote filename for xz command
That seems fixed, I ran into another issue.
> Makefile | 4 ++++
> copy-firmware.sh | 47 +++++++++++++++++++++++++++++++----------------
> 2 files changed, 35 insertions(+), 16 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index e1c362f..9a48471 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -11,3 +11,7 @@ check:
> install:
> mkdir -p $(DESTDIR)$(FIRMWAREDIR)
> ./copy-firmware.sh $(DESTDIR)$(FIRMWAREDIR)
> +
> +installcompress:
> + mkdir -p $(DESTDIR)$(FIRMWAREDIR)
> + ./copy-firmware.sh -C $(DESTDIR)$(FIRMWAREDIR)
> diff --git a/copy-firmware.sh b/copy-firmware.sh
> index 9b46b63..0dd2e5c 100755
> --- a/copy-firmware.sh
> +++ b/copy-firmware.sh
> @@ -6,6 +6,7 @@
>
> verbose=:
> prune=no
> +compress=no
>
> while test $# -gt 0; do
> case $1 in
> @@ -19,6 +20,11 @@ while test $# -gt 0; do
> shift
> ;;
>
> + -C | --compress)
> + compress=yes
> + shift
> + ;;
> +
> *)
> if test "x$destdir" != "x"; then
> echo "ERROR: unknown command-line options: $@"
> @@ -31,40 +37,49 @@ while test $# -gt 0; do
> esac
> done
>
> +if test "x$compress" = "xyes"; then
> + cmpxtn=.xz
> + grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
> + test -f "$f" || continue
> + $verbose "compressing $f"
> + xz -C crc32 "$f"
> + done
> +fi
This compresses the firmware in the local git directory. If you ever
run this once, it'll never compress any updated files if you run it
again and instead you'll get a bunch of "xz: <file>: File exists"
errors:
https://paste.centos.org/view/75c479da
That seems counter-intuitive and leaves everyone's git tree with a ton
of untracked files. Options I see:
1) Compress in place and copy, as-is in the patch, but then go back
and remove all the compressed files after the copy
2) Compress to a temp location, move to the destination from there
3) Use xz to directly output to the destination file instead of compress/copy
I kind of like 3. Thoughts?
josh
> +
> grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
> - test -f "$f" || continue
> - $verbose "copying file $f"
> - mkdir -p $destdir/$(dirname "$f")
> - cp -d "$f" $destdir/"$f"
> + test -f "$f$cmpxtn" || continue
> + $verbose "copying file $f$cmpxtn"
> + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> + cp -d "$f$cmpxtn" $destdir/"$f$cmpxtn"
> done
>
> grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
> - if test -L "$f"; then
> - test -f "$destdir/$f" && continue
> - $verbose "copying link $f"
> - mkdir -p $destdir/$(dirname "$f")
> + if test -L "$f$cmpxtn"; then
> + test -f "$destdir/$f$cmpxtn" && continue
> + $verbose "copying link $f$cmpxtn"
> + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> cp -d "$f" $destdir/"$f"
>
> if test "x$d" != "x"; then
> - target=`readlink "$f"`
> + target=`readlink "$f$cmpxtn"`
>
> if test "x$target" != "x$d"; then
> $verbose "WARNING: inconsistent symlink target: $target != $d"
> else
> if test "x$prune" != "xyes"; then
> - $verbose "WARNING: unneeded symlink detected: $f"
> + $verbose "WARNING: unneeded symlink detected: $f$cmpxtn"
> else
> - $verbose "WARNING: pruning unneeded symlink $f"
> - rm -f "$f"
> + $verbose "WARNING: pruning unneeded symlink $f$cmpxtn"
> + rm -f "$f$cmpxtn"
> fi
> fi
> else
> - $verbose "WARNING: missing target for symlink $f"
> + $verbose "WARNING: missing target for symlink $f$cmpxtn"
> fi
> else
> - $verbose "creating link $f -> $d"
> - mkdir -p $destdir/$(dirname "$f")
> - ln -sf "$d" "$destdir/$f"
> + $verbose "creating link $f$cmpxtn -> $d$cmpxtn"
> + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> + ln -sf "$d$cmpxtn" "$destdir/$f$cmpxtn"
> fi
> done
>
> --
> 2.29.2
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v2] Add support for compressing firmware in copy-firmware.sh
2021-02-08 13:47 ` Josh Boyer
@ 2021-02-08 13:50 ` Peter Robinson
0 siblings, 0 replies; 3+ messages in thread
From: Peter Robinson @ 2021-02-08 13:50 UTC (permalink / raw)
To: Josh Boyer; +Cc: Linux Firmware
On Mon, Feb 8, 2021 at 1:47 PM Josh Boyer <jwboyer@kernel.org> wrote:
>
> On Sat, Jan 23, 2021 at 5:26 AM Peter Robinson <pbrobinson@gmail.com> wrote:
> >
> > As of kernel 5.3 there's initial support for loading compressed firmware.
> > At this stage the only supported compression methis is "xz -C crc32" but
> > this option brings significant benefits.
> >
> > Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
> > ---
> >
> > v2: quote filename for xz command
>
> That seems fixed, I ran into another issue.
>
> > Makefile | 4 ++++
> > copy-firmware.sh | 47 +++++++++++++++++++++++++++++++----------------
> > 2 files changed, 35 insertions(+), 16 deletions(-)
> >
> > diff --git a/Makefile b/Makefile
> > index e1c362f..9a48471 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -11,3 +11,7 @@ check:
> > install:
> > mkdir -p $(DESTDIR)$(FIRMWAREDIR)
> > ./copy-firmware.sh $(DESTDIR)$(FIRMWAREDIR)
> > +
> > +installcompress:
> > + mkdir -p $(DESTDIR)$(FIRMWAREDIR)
> > + ./copy-firmware.sh -C $(DESTDIR)$(FIRMWAREDIR)
> > diff --git a/copy-firmware.sh b/copy-firmware.sh
> > index 9b46b63..0dd2e5c 100755
> > --- a/copy-firmware.sh
> > +++ b/copy-firmware.sh
> > @@ -6,6 +6,7 @@
> >
> > verbose=:
> > prune=no
> > +compress=no
> >
> > while test $# -gt 0; do
> > case $1 in
> > @@ -19,6 +20,11 @@ while test $# -gt 0; do
> > shift
> > ;;
> >
> > + -C | --compress)
> > + compress=yes
> > + shift
> > + ;;
> > +
> > *)
> > if test "x$destdir" != "x"; then
> > echo "ERROR: unknown command-line options: $@"
> > @@ -31,40 +37,49 @@ while test $# -gt 0; do
> > esac
> > done
> >
> > +if test "x$compress" = "xyes"; then
> > + cmpxtn=.xz
> > + grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
> > + test -f "$f" || continue
> > + $verbose "compressing $f"
> > + xz -C crc32 "$f"
> > + done
> > +fi
>
> This compresses the firmware in the local git directory. If you ever
> run this once, it'll never compress any updated files if you run it
> again and instead you'll get a bunch of "xz: <file>: File exists"
> errors:
>
> https://paste.centos.org/view/75c479da
>
> That seems counter-intuitive and leaves everyone's git tree with a ton
> of untracked files. Options I see:
>
> 1) Compress in place and copy, as-is in the patch, but then go back
> and remove all the compressed files after the copy
> 2) Compress to a temp location, move to the destination from there
> 3) Use xz to directly output to the destination file instead of compress/copy
>
> I kind of like 3. Thoughts?
Yes, I think that makes sense, I'd been looking at this from a distro
packaging perspective where it's generally run from the tarfile output
and then that is disgarded so less of an issue. Will take another pass
at it.
> josh
>
> > +
> > grep '^File:' WHENCE | sed -e's/^File: *//g' -e's/"//g' | while read f; do
> > - test -f "$f" || continue
> > - $verbose "copying file $f"
> > - mkdir -p $destdir/$(dirname "$f")
> > - cp -d "$f" $destdir/"$f"
> > + test -f "$f$cmpxtn" || continue
> > + $verbose "copying file $f$cmpxtn"
> > + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> > + cp -d "$f$cmpxtn" $destdir/"$f$cmpxtn"
> > done
> >
> > grep -E '^Link:' WHENCE | sed -e's/^Link: *//g' -e's/-> //g' | while read f d; do
> > - if test -L "$f"; then
> > - test -f "$destdir/$f" && continue
> > - $verbose "copying link $f"
> > - mkdir -p $destdir/$(dirname "$f")
> > + if test -L "$f$cmpxtn"; then
> > + test -f "$destdir/$f$cmpxtn" && continue
> > + $verbose "copying link $f$cmpxtn"
> > + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> > cp -d "$f" $destdir/"$f"
> >
> > if test "x$d" != "x"; then
> > - target=`readlink "$f"`
> > + target=`readlink "$f$cmpxtn"`
> >
> > if test "x$target" != "x$d"; then
> > $verbose "WARNING: inconsistent symlink target: $target != $d"
> > else
> > if test "x$prune" != "xyes"; then
> > - $verbose "WARNING: unneeded symlink detected: $f"
> > + $verbose "WARNING: unneeded symlink detected: $f$cmpxtn"
> > else
> > - $verbose "WARNING: pruning unneeded symlink $f"
> > - rm -f "$f"
> > + $verbose "WARNING: pruning unneeded symlink $f$cmpxtn"
> > + rm -f "$f$cmpxtn"
> > fi
> > fi
> > else
> > - $verbose "WARNING: missing target for symlink $f"
> > + $verbose "WARNING: missing target for symlink $f$cmpxtn"
> > fi
> > else
> > - $verbose "creating link $f -> $d"
> > - mkdir -p $destdir/$(dirname "$f")
> > - ln -sf "$d" "$destdir/$f"
> > + $verbose "creating link $f$cmpxtn -> $d$cmpxtn"
> > + mkdir -p $destdir/$(dirname "$f$cmpxtn")
> > + ln -sf "$d$cmpxtn" "$destdir/$f$cmpxtn"
> > fi
> > done
> >
> > --
> > 2.29.2
> >
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-02-08 13:51 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-23 10:26 [PATCH v2] Add support for compressing firmware in copy-firmware.sh Peter Robinson
2021-02-08 13:47 ` Josh Boyer
2021-02-08 13:50 ` Peter Robinson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).