Question: v20231111 adds rdfind dependency: why not at ball build time?

Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Hello.

I see my Linux distro updating its package, and adding hard
rdfind(1) dependency because that copy-*.sh file uses it (and even
added an error-out if not available).

That made me wonder two times: first of all i like that duplicates
vanish, but that is not "a killer".  People downloaded the ball,
and then they get thrown out because the ball is to be slimmed on
their box.  And then, why is the ball slimmed-down on all those
myriads of user boxes after having been downloaded, instead of
once when the ball is created during release?  That is so strange,
you know?

Thank you for all your effort.  (Not subscribed.)

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Mario Limonciello]

On 11/13/2023 15:53, Steffen Nurpmeso wrote:
> Hello.
> 
> I see my Linux distro updating its package, and adding hard
> rdfind(1) dependency because that copy-*.sh file uses it (and even
> added an error-out if not available).
> 
> That made me wonder two times: first of all i like that duplicates
> vanish, but that is not "a killer".  People downloaded the ball,
> and then they get thrown out because the ball is to be slimmed on
> their box.  And then, why is the ball slimmed-down on all those
> myriads of user boxes after having been downloaded, instead of
> once when the ball is created during release?  That is so strange,
> you know?

I believe it's because of the way it's created and signed it's not 
possible to run a build action before creation.

Another possibility is that we can just run rdfind on the repo and store 
a bunch of symlinks but I worry that if a vendor submits a binary that
used to be the same for two products but no longer is it will lead to 
unexpected results.

> 
> Thank you for all your effort.  (Not subscribed.)
> 
> --steffen
> |
> |Der Kragenbaer,                The moon bear,
> |der holt sich munter           he cheerfully and one by one
> |einen nach dem anderen runter  wa.ks himself off
> |(By Robert Gernhardt)
> 

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Hello!

Mario Limonciello wrote in
 <73529434-9b82-4131-8cbd-05eff75537db@amd.com>:
 |On 11/13/2023 15:53, Steffen Nurpmeso wrote:
 |> I see my Linux distro updating its package, and adding hard
 |> rdfind(1) dependency because that copy-*.sh file uses it (and even
 |> added an error-out if not available).
 |> 
 |> That made me wonder two times: first of all i like that duplicates
 |> vanish, but that is not "a killer".  People downloaded the ball,
 |> and then they get thrown out because the ball is to be slimmed on
 |> their box.  And then, why is the ball slimmed-down on all those
 |> myriads of user boxes after having been downloaded, instead of
 |> once when the ball is created during release?  That is so strange,
 |> you know?
 |
 |I believe it's because of the way it's created and signed it's not 
 |possible to run a build action before creation.

That is bitter.  (And cannot be changed??)

 |Another possibility is that we can just run rdfind on the repo and store 
 |a bunch of symlinks but I worry that if a vendor submits a binary that
 |used to be the same for two products but no longer is it will lead to 
 |unexpected results.

If you could run it at ball-build time then you could simply
create the entire installation plan as a shell script, instead of
running these complicated pipes on the user boxes.
I mean the ball is so huge and few kilobytes shell script does
then not matter.  (Not that i think i say something noone ever has
thought of, you know -- it just all seems so, hm, excessive.)

 |> Thank you for all your effort.  (Not subscribed.)

Yes.
Ciao,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20231117011925.PPRKHhpq@steffen%sdaoden.eu>:
 |Mario Limonciello wrote in
 | <73529434-9b82-4131-8cbd-05eff75537db@amd.com>:
 ||On 11/13/2023 15:53, Steffen Nurpmeso wrote:
 ||> I see my Linux distro updating its package, and adding hard
 ||> rdfind(1) dependency because that copy-*.sh file uses it (and even
 ||> added an error-out if not available).
 ||> 
 ||> That made me wonder two times: first of all i like that duplicates
 ||> vanish, but that is not "a killer".  People downloaded the ball,
 ||> and then they get thrown out because the ball is to be slimmed on
 ||> their box.  And then, why is the ball slimmed-down on all those
 ||> myriads of user boxes after having been downloaded, instead of
 ||> once when the ball is created during release?  That is so strange,
 ||> you know?
 ||
 ||I believe it's because of the way it's created and signed it's not 
 ||possible to run a build action before creation.
 |
 |That is bitter.  (And cannot be changed??)
 |
 ||Another possibility is that we can just run rdfind on the repo and store 
 ||a bunch of symlinks but I worry that if a vendor submits a binary that
 ||used to be the same for two products but no longer is it will lead to 
 ||unexpected results.
 |
 |If you could run it at ball-build time then you could simply
 |create the entire installation plan as a shell script, instead of
 |running these complicated pipes on the user boxes.
 |I mean the ball is so huge and few kilobytes shell script does
 |then not matter.  (Not that i think i say something noone ever has
 |thought of, you know -- it just all seems so, hm, excessive.)

Now it goes too far, but i have a simple backup script (for those
few resource files not under git control), and it places all files
under /root/hosts/$HOSTNAME/backup.d/ and creates a .restore.dat
like

  . root root 644 _etc_adjtime /etc/adjtime
...
  . root users 640 _etc_krb5_conf /etc/krb5.conf
...
  @ root root 777 _etc_localtime /etc/localtime
...

etc etc, with / being dir (first), . files and @ symlinks.
Ie what i mean is, why store symlinks in the ball at all?  Simply
create an installation script via rdfind at ball-build time which
then does the necessary things.

 ||> Thank you for all your effort.  (Not subscribed.)
 |
 |Yes.
 |Ciao,

Greetings.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Thank you very much for making rdfind an optional thing!

I still have not understood why this complicated mechanism is
applied twice, for both creating a distribution tarball and then
again on the user box, instead of maybe (if it really has to be
done that hard) doing a final find(1) run during tarball creation,
and creating a schedule what to cp(1) and what to ln(1) -s (and
where to, of course), however.

But i wanted to say thank you, and then i was wondering why the
Linux kernel uses --ultra -22 for zstd(1), whereas Linux-firmware
uses a very, very low number.  Also note that zstd(1) -T0 works
regardless of whether SMP is compiled in or not, last time
i looked, it is simply ignored without.  And it is very helpful,
especially with --ultra -22 and on a modern box.

Thanks again, and
Ciao,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
|
| Only in December: lightful Dubai COP28 Narendra Modi quote:
|  A small part of humanity has ruthlessly exploited nature.
|  But the entire humanity is bearing the cost of it,
|  especially the inhabitants of the Global South.
|  The selfishness of a few will lead the world into darkness,
|  not just for themselves but for the entire world.
|  [Christians might think of Revelation 11:18
|    The nations were angry, and your wrath has come[.]
|    [.]for destroying those who destroy the earth.
|   But i find the above more kind, and much friendlier]

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Josh Boyer]

On Sat, Dec 16, 2023 at 5:36 PM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
>
> Thank you very much for making rdfind an optional thing!
>
> I still have not understood why this complicated mechanism is
> applied twice, for both creating a distribution tarball and then
> again on the user box, instead of maybe (if it really has to be
> done that hard) doing a final find(1) run during tarball creation,
> and creating a schedule what to cp(1) and what to ln(1) -s (and
> where to, of course), however.

The tarball is created automatically by kernel.org infrastructure
based on a signed git tag.  The maintainers do not have access to the
backend that creates the tarball, so we can't run rdfind during
creation.

> But i wanted to say thank you, and then i was wondering why the
> Linux kernel uses --ultra -22 for zstd(1), whereas Linux-firmware
> uses a very, very low number.  Also note that zstd(1) -T0 works
> regardless of whether SMP is compiled in or not, last time
> i looked, it is simply ignored without.  And it is very helpful,
> especially with --ultra -22 and on a modern box.

I'm honestly not sure.  You'd have to ask the author of the commit
that introduced zstd support as to why they chose those numbers.

josh

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Hello.

Josh Boyer wrote in
 <CA+5PVA7fCsgSAYDcMq4NSC4u1ax0f=6WmHo76LZsBRe3gvt5og@mail.gmail.com>:
 |On Sat, Dec 16, 2023 at 5:36 PM Steffen Nurpmeso <steffen@sdaoden.eu> \
 |wrote:
 |> Thank you very much for making rdfind an optional thing!
 |>
 |> I still have not understood why this complicated mechanism is
 |> applied twice, for both creating a distribution tarball and then
 |> again on the user box, instead of maybe (if it really has to be
 |> done that hard) doing a final find(1) run during tarball creation,
 |> and creating a schedule what to cp(1) and what to ln(1) -s (and
 |> where to, of course), however.
 |
 |The tarball is created automatically by kernel.org infrastructure
 |based on a signed git tag.  The maintainers do not have access to the
 |backend that creates the tarball, so we can't run rdfind during
 |creation.

Ah!  Now i understand what Mario Limonciello said!

Hm.  Are you interested in a patch that uses the cksum(1) that is
installed everywhere (except that stdout whitespace needs to be
normalized no Sun/Solaris)?  Ie, simply find(1) all files, and
place their cksum(1) with path in a file, sort(1) that, and then
work it linewise, causing identical follow-ups to become symbolic
links?  (Or rather, add another cmp(1) to be absolutely sure?)
Then store that file in the tarball and work the recipe on user
boxes?

 |> But i wanted to say thank you, and then i was wondering why the
 |> Linux kernel uses --ultra -22 for zstd(1), whereas Linux-firmware
 |> uses a very, very low number.  Also note that zstd(1) -T0 works
 |> regardless of whether SMP is compiled in or not, last time
 |> i looked, it is simply ignored without.  And it is very helpful,
 |> especially with --ultra -22 and on a modern box.
 |
 |I'm honestly not sure.  You'd have to ask the author of the commit
 |that introduced zstd support as to why they chose those numbers.

Thanks for the answer anyway!

 |josh
 --End of <CA+5PVA7fCsgSAYDcMq4NSC4u1ax0f=6WmHo76LZsBRe3gvt5og@mail.gmail\
 .com>

Ok.  I have *never* tried it with rdfind, so the actual symbolic
link may not have the desired value (see commented out thing), but
isn't this essentially what is desired?
It only uses cksum, find, sort, etc, and can be run when creating
the tarball: on the target host no more utilities have to run at
all, simply unpack the tarball??
(Except: compression.  make install: should simply find(1) -type
f and call compressor if so desired??)

--- Makefile.orig	2024-01-02 22:30:37.367929396 +0100
+++ Makefile	2024-01-02 23:11:28.864549352 +0100
@@ -14,7 +14,7 @@ check:
 
 dist:
 	@mkdir -p release dist
-	./copy-firmware.sh release
+	DIST_CREATE=y ./copy-firmware.sh release
 	@TARGET=linux-firmware_`git describe`.tar.gz; \
 	cd release && tar -czf ../dist/$${TARGET} *; \
 	echo "Created dist/$${TARGET}"
--- copy-firmware.sh.orig	2024-01-02 22:30:43.584595944 +0100
+++ copy-firmware.sh	2024-01-02 23:12:17.457881758 +0100
@@ -11,6 +11,8 @@ compress=cat
 compext=
 skip_dedup=0
 
+: ${dist_create:=$DIST_CREATE}
+
 while test $# -gt 0; do
     case $1 in
         -v | --verbose)
@@ -40,7 +42,7 @@ while test $# -gt 0; do
                 exit 1
             fi
             # shellcheck disable=SC2209
-            compress="zstd --compress --quiet --stdout"
+            compress="zstd -T0 --ultra -22 --compress --quiet --stdout"
             compext=".zst"
             shift
             ;;
@@ -75,13 +77,6 @@ if [ -z "$destdir" ]; then
 	exit 1
 fi
 
-if ! command -v rdfind >/dev/null; then
-	if [ "$skip_dedup" != 1 ]; then
-    		echo "ERROR: rdfind is not installed.  Pass --ignore-duplicates to skip deduplication"
-		exit 1
-	fi
-fi
-
 # shellcheck disable=SC2162 # file/folder name can include escaped symbols
 grep -E '^(RawFile|File):' WHENCE | sed -E -e 's/^(RawFile|File): */\1 /;s/"//g' | while read k f; do
     test -f "$f" || continue
@@ -89,20 +84,34 @@ grep -E '^(RawFile|File):' WHENCE | sed
     $verbose "copying/compressing file $f$compext"
     if test "$compress" != "cat" && test "$k" = "RawFile"; then
         $verbose "compression will be skipped for file $f"
-        cat "$f" > "$destdir/$f"
+        cp "$f" "$destdir/$f"
+    elif [ "$compress" = cat ]; then
+        cp "$f" "$destdir/$f"
     else
         $compress "$f" > "$destdir/$f$compext"
     fi
 done
 
-if [ "$skip_dedup" != 1 ] ; then
+if [ "$skip_dedup" != 1 ] && [ -n "$DIST_CREATE" ]; then
 	$verbose "Finding duplicate files"
-	rdfind -makesymlinks true -makeresultsfile false "$destdir" >/dev/null
-	find "$destdir" -type l | while read -r l; do
-		target="$(realpath "$l")"
-		$verbose "Correcting path for $l"
-		ln -fs "$(realpath --relative-to="$(dirname "$(realpath -s "$l")")" "$target")" "$l"
-	done
+    (
+    cd "$destdir" && find . -type f | xargs cksum | sort | {
+        ls= lf=
+        while read s1 s2 f; do
+            s="$s1 $s2"
+            #$verbose $s $f
+            if [ "$s" = "$ls" ] && cmp "$lf" "$f"; then
+                $verbose 'duplicate '"${lf##*/}" "${f#./*}"
+                rm -f "$f"
+                #ln -s "${lf#./*}" "${f#./*}"
+                ln -s "${lf##*/}" "${f#./*}"
+            else
+                ls=$s
+                lf=$f
+            fi
+        done
+    }
+    )
 fi
 
 # shellcheck disable=SC2162 # file/folder name can include escaped symbols

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Hm, hello again.

Steffen Nurpmeso wrote in
 <20240102221621.YaA1nlef@steffen%sdaoden.eu>:
 |Josh Boyer wrote in
 | <CA+5PVA7fCsgSAYDcMq4NSC4u1ax0f=6WmHo76LZsBRe3gvt5og@mail.gmail.com>:
 ||On Sat, Dec 16, 2023 at 5:36 PM Steffen Nurpmeso <steffen@sdaoden.eu> \
 ||wrote:
 ||> Thank you very much for making rdfind an optional thing!
 ||>
 ||> I still have not understood why this complicated mechanism is

It is a _terrible_ and terribly redundant build system.
But that is not my problem, i will not look into what these
build_packages.py things do...
However my thing from yesterday is wrong in so far as the
duplicates are only removed when run from "make dist", which is at
least once or twice not enough, then.

So please this one simply hammers always (unless dedup is not
desired).  Also it uses -f arg to ln(1) for whatever reason.
Otherwise identical.

Again i personally would instead do copy_firmware.sh once, and do
not create any links during that run at all.  Instead i would
place those with the target in a text file to be shipped along.
At install time (i would look in an "updates" path??) i would run
that recipe and create according (compressed?) links as necessary.
(If "updates" contains file, these have to be taken into account
when doing so, possibly skipping a link, but doing a copy.)

I do no understand how copy-firmware.sh can currently work on
already linked data "when there are updates to take into account"
(and there is a mismatch).

Anyhow.  This diff should also work on github, or wherever "make
dist" is run, since cksum, sort, cp, find and xargs should all be
available by default.

Ciao from Germany!

--- copy-firmware.sh.orig	2024-01-02 22:30:43.000000000 +0100
+++ copy-firmware.sh	2024-01-03 20:38:58.157253235 +0100
@@ -40,7 +40,7 @@ while test $# -gt 0; do
                 exit 1
             fi
             # shellcheck disable=SC2209
-            compress="zstd --compress --quiet --stdout"
+            compress="zstd -T0 --ultra -22 --compress --quiet --stdout"
             compext=".zst"
             shift
             ;;
@@ -75,13 +75,6 @@ if [ -z "$destdir" ]; then
 	exit 1
 fi
 
-if ! command -v rdfind >/dev/null; then
-	if [ "$skip_dedup" != 1 ]; then
-    		echo "ERROR: rdfind is not installed.  Pass --ignore-duplicates to skip deduplication"
-		exit 1
-	fi
-fi
-
 # shellcheck disable=SC2162 # file/folder name can include escaped symbols
 grep -E '^(RawFile|File):' WHENCE | sed -E -e 's/^(RawFile|File): */\1 /;s/"//g' | while read k f; do
     test -f "$f" || continue
@@ -89,20 +82,34 @@ grep -E '^(RawFile|File):' WHENCE | sed
     $verbose "copying/compressing file $f$compext"
     if test "$compress" != "cat" && test "$k" = "RawFile"; then
         $verbose "compression will be skipped for file $f"
-        cat "$f" > "$destdir/$f"
+        cp "$f" "$destdir/$f"
+    elif [ "$compress" = cat ]; then
+        cp "$f" "$destdir/$f"
     else
         $compress "$f" > "$destdir/$f$compext"
     fi
 done
 
-if [ "$skip_dedup" != 1 ] ; then
+if [ "$skip_dedup" != 1 ]; then
 	$verbose "Finding duplicate files"
-	rdfind -makesymlinks true -makeresultsfile false "$destdir" >/dev/null
-	find "$destdir" -type l | while read -r l; do
-		target="$(realpath "$l")"
-		$verbose "Correcting path for $l"
-		ln -fs "$(realpath --relative-to="$(dirname "$(realpath -s "$l")")" "$target")" "$l"
-	done
+    (
+    cd "$destdir" && find . -type f | xargs cksum | sort | {
+        ls= lf=
+        while read s1 s2 f; do
+            s="$s1 $s2"
+            #$verbose $s $f
+            if [ "$s" = "$ls" ] && cmp "$lf" "$f"; then
+                $verbose 'duplicate '"${lf##*/}" "${f#./*}"
+                rm -f "$f"
+                #ln -s "${lf#./*}" "${f#./*}"
+                ln -sf "${lf##*/}" "${f#./*}"
+            else
+                ls=$s
+                lf=$f
+            fi
+        done
+    }
+    )
 fi
 
 # shellcheck disable=SC2162 # file/folder name can include escaped symbols

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question: v20231111 adds rdfind dependency: why not at ball build time?

[Steffen Nurpmeso]

Hello again,

i though i ask again after a month has passed.

Steffen Nurpmeso wrote in
 <20240103194300.jKoOqQ-g@steffen%sdaoden.eu>:
 |Steffen Nurpmeso wrote in
 | <20240102221621.YaA1nlef@steffen%sdaoden.eu>:
 ||Josh Boyer wrote in
 || <CA+5PVA7fCsgSAYDcMq4NSC4u1ax0f=6WmHo76LZsBRe3gvt5og@mail.gmail.com>:
 |||On Sat, Dec 16, 2023 at 5:36 PM Steffen Nurpmeso <steffen@sdaoden.eu> \
 |||wrote:
 |||> Thank you very much for making rdfind an optional thing!
 ...
 |Anyhow.  This diff should also work on github, or wherever "make
 |dist" is run, since cksum, sort, cp, find and xargs should all be
 |available by default.

Is there anything i can do to bring this forward?
I mean, copy-firmware.sh would loose the (now soft) dependency to
rdfind, replacing it with all-POSIX and by-default programs.
I could remove the zstd(1) adjustment (which brings linux-firmware
into alignment with the linux kernel), other than that it seems to
be fine?  Really no interest in keeping the functionality but
without rdfind that is needed for no other purpose?

Ciao already here,

 |--- copy-firmware.sh.orig     2024-01-02 22:30:43.000000000 +0100
 |+++ copy-firmware.sh  2024-01-03 20:38:58.157253235 +0100
 |@@ -40,7 +40,7 @@ while test $# -gt 0; do
 |                 exit 1
 |             fi
 |             # shellcheck disable=SC2209
 |-            compress="zstd --compress --quiet --stdout"
 |+            compress="zstd -T0 --ultra -22 --compress --quiet --stdout"
 |             compext=".zst"
 |             shift
 |             ;;
 |@@ -75,13 +75,6 @@ if [ -z "$destdir" ]; then
 |  exit 1
 | fi
 | 
 |-if ! command -v rdfind >/dev/null; then
 |-     if [ "$skip_dedup" != 1 ]; then
 |-             echo "ERROR: rdfind is not installed.  Pass --ignore-dupli\
 |cates to skip deduplication"
 |-             exit 1
 |-     fi
 |-fi
 |-
 | # shellcheck disable=SC2162 # file/folder name can include escaped symbols
 | grep -E '^(RawFile|File):' WHENCE | sed -E -e 's/^(RawFile|File): \
 | */\1 /;s/"//g' | while read k f; do
 |     test -f "$f" || continue
 |@@ -89,20 +82,34 @@ grep -E '^(RawFile|File):' WHENCE | sed
 |     $verbose "copying/compressing file $f$compext"
 |     if test "$compress" != "cat" && test "$k" = "RawFile"; then
 |         $verbose "compression will be skipped for file $f"
 |-        cat "$f" > "$destdir/$f"
 |+        cp "$f" "$destdir/$f"
 |+    elif [ "$compress" = cat ]; then
 |+        cp "$f" "$destdir/$f"
 |     else
 |         $compress "$f" > "$destdir/$f$compext"
 |     fi
 | done
 | 
 |-if [ "$skip_dedup" != 1 ] ; then
 |+if [ "$skip_dedup" != 1 ]; then
 |  $verbose "Finding duplicate files"
 |-     rdfind -makesymlinks true -makeresultsfile false "$destdir" >/dev/\
 |null
 |-     find "$destdir" -type l | while read -r l; do
 |-             target="$(realpath "$l")"
 |-             $verbose "Correcting path for $l"
 |-             ln -fs "$(realpath --relative-to="$(dirname "$(realpath \
 |-s "$l")")" "$target")" "$l"
 |-     done
 |+    (
 |+    cd "$destdir" && find . -type f | xargs cksum | sort | {
 |+        ls= lf=
 |+        while read s1 s2 f; do
 |+            s="$s1 $s2"
 |+            #$verbose $s $f
 |+            if [ "$s" = "$ls" ] && cmp "$lf" "$f"; then
 |+                $verbose 'duplicate '"${lf##*/}" "${f#./*}"
 |+                rm -f "$f"
 |+                #ln -s "${lf#./*}" "${f#./*}"
 |+                ln -sf "${lf##*/}" "${f#./*}"
 |+            else
 |+                ls=$s
 |+                lf=$f
 |+            fi
 |+        done
 |+    }
 |+    )
 | fi
 | 
 | # shellcheck disable=SC2162 # file/folder name can include escaped symbols

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)