[Buildroot] [PATCH v3 1/2] package/uutils-coreutils: new package

[Buildroot] [PATCH v3 1/2] package/uutils-coreutils: new package

[Sebastian Weyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 9471 bytes --]

This package is an implementation of coreutils written completely in rust.

Using the gnu-coreutils-testsuite on this implementation of coreutils on
the target is sadly pretty difficult since it is very memory and
computation intensive.
In order to compare the two implementations I wrote a shell script that
just extracted the command line options of every command that is also
provided by the coreutils package.

These commands are the following: base64,
cat, chgrp, chmod, chown, cp, date, dd, df, dir, echo, false, join, kill,
link, ln, ls, mkdir, mknod, mktemp, mv, nice, printenv, pwd, rm, rmdir,
sleep, stty, sync, touch, true, uname, vdir

The changes I noticed were that the command "stty" is missing from this
implementation of coreutils. Furthermore I noticed the following
missing/changed arguments:

chmod   -> --silent becomes --quiet
cp      -> -Z option missing
date    -> --utc becomes --universal
dir     -> -D, --dired, --hyperlink missing
ln      -> -d, -F, --directory missing
ls      -> -D, --dired, --hyperlink, --indicator missing
mkdir   -> -Z, --context missing
mknod   -> -Z, --context missing
mv      -> -Z, --context missing
vdir    -> -D, --dired, --hyperlink, --indicator missing
uname   -> -i, --hardware-platform, -p, --processor missing

In this list I didn't include arguments that were newly added since they
wouldn't introduce breaking. I can not make any claim about the exact
behaviour of each of the commands with their respective arguments. The
maintainers of the project provide a page in their documentation,
detailling the compatibility with the gnu-coreutils-testsuite:
https://uutils.github.io/user/test_coverage.html

Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
---
The installation (creating the symlinks pointing to the main binary) is done
using the project's Makefile. A patch was added to remove the build dependency
of the install step in said Makefile. Since install is always called after
build, this avoids the install step rebuilding the application.

The optional dependency in package/busybox/busybox.mk is necessary so that
when uutils-coreutils is selected, it will be built before busybox. Then
busybox will see the symlinks that are already placed in target/bin and will
not recreate them (noclobber).

v2:
- created dependency between coreutils and uutils-coreutils so one can't
- be activated at the same time as the other
- Fixed my name
- Added a patch that removes the generation of autocompletions for some
  shells
- using cargo infrastructure for building now. Installing is still done
  using the project's Makefile
- implemented release and debug profiles depending on what is chosen for
  BR2_DEBUG_ENABLE

v3:
Added comment explaining UUTILS_COREUTILS_INSTALL_TARGET_CMDS step and
fixed PROFILE variable in UUTILS_COREUTILS_MAKE_OPTS

Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
---
 DEVELOPERS                                    |  3 ++
 package/Config.in                             |  1 +
 package/busybox/busybox.mk                    |  1 +
 package/coreutils/Config.in                   |  1 +
 ...ve-dependency-on-build-during-instal.patch | 30 ++++++++++++++++
 package/uutils-coreutils/Config.in            | 12 +++++++
 .../uutils-coreutils/uutils-coreutils.hash    |  3 ++
 package/uutils-coreutils/uutils-coreutils.mk  | 35 +++++++++++++++++++
 8 files changed, 86 insertions(+)
 create mode 100644 package/uutils-coreutils/0001-GNUMakefile-remove-dependency-on-build-during-instal.patch
 create mode 100644 package/uutils-coreutils/Config.in
 create mode 100644 package/uutils-coreutils/uutils-coreutils.hash
 create mode 100644 package/uutils-coreutils/uutils-coreutils.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index b6d288c54f..2d23667953 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2583,6 +2583,9 @@ F:	support/misc/toolchainfile.cmake.in
 N:	Sam Voss <sam.voss@gmail.com>
 F:	package/ripgrep/
 
+N:	Sebastian Weyer <sebastian.weyer@smile.fr>
+F:	package/uutils-coreutils
+
 N:	Sébastien Szymanski <sebastien.szymanski@armadeus.com>
 F:	package/mmc-utils/
 F:	package/python-flask-jsonrpc/
diff --git a/package/Config.in b/package/Config.in
index 0f8dab3e71..79bbe0b44d 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2725,6 +2725,7 @@ menu "System tools"
 	source "package/tpm2-totp/Config.in"
 	source "package/unscd/Config.in"
 	source "package/util-linux/Config.in"
+	source "package/uutils-coreutils/Config.in"
 	source "package/watchdog/Config.in"
 	source "package/watchdogd/Config.in"
 	source "package/xdg-dbus-proxy/Config.in"
diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 1633ca525d..d8000ddbb4 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -68,6 +68,7 @@ BUSYBOX_DEPENDENCIES = \
 	$(if $(BR2_PACKAGE_UNZIP),unzip) \
 	$(if $(BR2_PACKAGE_USBUTILS),usbutils) \
 	$(if $(BR2_PACKAGE_UTIL_LINUX),util-linux) \
+	$(if $(BR2_PACKAGE_UUTILS_COREUTILS),uutils-coreutils) \
 	$(if $(BR2_PACKAGE_VIM),vim) \
 	$(if $(BR2_PACKAGE_WATCHDOG),watchdog) \
 	$(if $(BR2_PACKAGE_WGET),wget) \
diff --git a/package/coreutils/Config.in b/package/coreutils/Config.in
index 705013bae0..9c1af1dc25 100644
--- a/package/coreutils/Config.in
+++ b/package/coreutils/Config.in
@@ -3,6 +3,7 @@ config BR2_PACKAGE_COREUTILS
 	depends on BR2_USE_WCHAR
 	depends on BR2_USE_MMU # fork()
 	depends on BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
+	depends on !BR2_PACKAGE_UUTILS_COREUTILS
 	help
 	  All of the basic file/text/shell utilities. These are the
 	  core utilities which are expected to exist on every system.
diff --git a/package/uutils-coreutils/0001-GNUMakefile-remove-dependency-on-build-during-instal.patch b/package/uutils-coreutils/0001-GNUMakefile-remove-dependency-on-build-during-instal.patch
new file mode 100644
index 0000000000..a2e4a17ac0
--- /dev/null
+++ b/package/uutils-coreutils/0001-GNUMakefile-remove-dependency-on-build-during-instal.patch
@@ -0,0 +1,30 @@
+From ed387083221501aaf5d8752aab83669174e0bb43 Mon Sep 17 00:00:00 2001
+From: Sebastian Weyer <sebastian.weyer@smile.fr>
+Date: Thu, 16 Mar 2023 15:57:13 +0100
+Subject: [PATCH] GNUMakefile: remove dependency on build during install
+
+When building using buildroot, we only call install after having built
+already, therefore we don't want to have to rebuild.
+
+Upstream status: not applicable; Buildroot specific
+Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
+---
+ GNUmakefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/GNUmakefile b/GNUmakefile
+index b242bc8ce..faecf420b 100644
+--- a/GNUmakefile
++++ b/GNUmakefile
+@@ -334,7 +334,7 @@ clean:
+ distclean: clean
+ 	$(CARGO) clean $(CARGOFLAGS) && $(CARGO) update $(CARGOFLAGS)
+ 
+-install: build
++install:
+ 	mkdir -p $(INSTALLDIR_BIN)
+ ifeq (${MULTICALL}, y)
+ 	$(INSTALL) $(BUILDDIR)/coreutils $(INSTALLDIR_BIN)/$(PROG_PREFIX)coreutils
+-- 
+2.25.1
+
diff --git a/package/uutils-coreutils/Config.in b/package/uutils-coreutils/Config.in
new file mode 100644
index 0000000000..97ddd4b096
--- /dev/null
+++ b/package/uutils-coreutils/Config.in
@@ -0,0 +1,12 @@
+config BR2_PACKAGE_UUTILS_COREUTILS
+	bool "uutils-coreutils"
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_RUSTC
+	depends on BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
+	help
+	  uutils is an attempt at writing universal (as in
+	  cross-platform) CLI utilities in Rust. While all programs have
+	  been implemented, some options might be missing or different
+	  behavior might be experienced.
+
+	  https://github.com/uutils/coreutils
diff --git a/package/uutils-coreutils/uutils-coreutils.hash b/package/uutils-coreutils/uutils-coreutils.hash
new file mode 100644
index 0000000000..ca90a0c9f2
--- /dev/null
+++ b/package/uutils-coreutils/uutils-coreutils.hash
@@ -0,0 +1,3 @@
+# Locally generated
+sha256  5bc773bcbc66851aa17979df44224c66c0b5323044c3c9cefb925b44ee9cd81b  uutils-coreutils-0.0.17.tar.gz
+sha256  a836eb3360d0e80f6a1620495543ad2a4e20e4cf72af89ff528fccf9e63be81e  LICENSE
diff --git a/package/uutils-coreutils/uutils-coreutils.mk b/package/uutils-coreutils/uutils-coreutils.mk
new file mode 100644
index 0000000000..a0ea06ff5f
--- /dev/null
+++ b/package/uutils-coreutils/uutils-coreutils.mk
@@ -0,0 +1,35 @@
+################################################################################
+#
+# uutils-coreutils
+#
+################################################################################
+
+UUTILS_COREUTILS_VERSION = 0.0.17
+UUTILS_COREUTILS_SITE = $(call github,uutils,coreutils,$(UUTILS_COREUTILS_VERSION))
+UUTILS_COREUTILS_LICENSE = MIT
+UUTILS_COREUTILS_LICENSE_FILES = LICENSE
+
+ifneq ($(BR2_ENABLE_DEBUG),y)
+UUTILS_COREUTILS_PROFILE = release
+else
+UUTILS_COREUTILS_PROFILE = debug
+endif
+
+UUTILS_COREUTILS_MAKE_OPTS = \
+	PROFILE=$(UUTILS_COREUTILS_PROFILE) \
+	MULTICALL=y
+
+UUTILS_COREUTILS_CARGO_BUILD_OPTS = --features unix
+
+#We are using the project's Makefile to install the application
+#If we were to use the cargo infrastructure, we can only copy the multicall
+#binary into the target folder. The call to the target's Makefile will do that
+#and also automatically create the required symlinks
+define UUTILS_COREUTILS_INSTALL_TARGET_CMDS
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		$(UUTILS_COREUTILS_MAKE_OPTS) -C $(@D) \
+		BUILDDIR=$(@D)/target/$(RUSTC_TARGET_NAME)/$(UUTILS_COREUTILS_PROFILE) \
+		PREFIX=$(TARGET_DIR) install
+endef
+
+$(eval $(cargo-package))
-- 
2.25.1


[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 2/2] package/uutils-coreutils: remove creation of autocompletions

[Sebastian Weyer]

Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
---
 ...emove-generation-of-auto-completions.patch | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 package/uutils-coreutils/0002-GNUmakefile-remove-generation-of-auto-completions.patch

diff --git a/package/uutils-coreutils/0002-GNUmakefile-remove-generation-of-auto-completions.patch b/package/uutils-coreutils/0002-GNUmakefile-remove-generation-of-auto-completions.patch
new file mode 100644
index 0000000000..524b5b05da
--- /dev/null
+++ b/package/uutils-coreutils/0002-GNUmakefile-remove-generation-of-auto-completions.patch
@@ -0,0 +1,45 @@
+From d212a5bdfe4207316c4410191e61b4d52c3c633a Mon Sep 17 00:00:00 2001
+From: Sebastian Weyer <sebastian.weyer@smile.fr>
+Date: Thu, 16 Mar 2023 15:58:25 +0100
+Subject: [PATCH] GNUmakefile: remove generation of auto-completions
+
+The install step of GNUmakefile by default includes the generation of
+autocompletions for bash, zsh and fish.
+
+There are 2 problems with this:
+
+1. In a buildroot context we don't always have bash, zsh or fish
+   installed and therefore we probably don't need these anyways
+
+2. These autocompletions are generated using the compiled binary. Therefore
+   if desired it should be generated on the target itself. Otherwise it might
+   not be executable on the host.
+
+Upstream status: not applicable; Buildroot specific
+Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
+---
+ GNUmakefile | 8 --------
+ 1 file changed, 8 deletions(-)
+
+diff --git a/GNUmakefile b/GNUmakefile
+index faecf420b..1c44cb3c6 100644
+--- a/GNUmakefile
++++ b/GNUmakefile
+@@ -346,14 +346,6 @@ else
+ 		$(INSTALL) $(BUILDDIR)/$(prog) $(INSTALLDIR_BIN)/$(PROG_PREFIX)$(prog);)
+ 	$(if $(findstring test,$(INSTALLEES)), $(INSTALL) $(BUILDDIR)/test $(INSTALLDIR_BIN)/$(PROG_PREFIX)[)
+ endif
+-	mkdir -p $(DESTDIR)$(DATAROOTDIR)/zsh/site-functions
+-	mkdir -p $(DESTDIR)$(DATAROOTDIR)/bash-completion/completions
+-	mkdir -p $(DESTDIR)$(DATAROOTDIR)/fish/vendor_completions.d
+-	$(foreach prog, $(INSTALLEES), \
+-		$(BUILDDIR)/coreutils completion $(prog) zsh > $(DESTDIR)$(DATAROOTDIR)/zsh/site-functions/_$(PROG_PREFIX)$(prog); \
+-		$(BUILDDIR)/coreutils completion $(prog) bash > $(DESTDIR)$(DATAROOTDIR)/bash-completion/completions/$(PROG_PREFIX)$(prog); \
+-		$(BUILDDIR)/coreutils completion $(prog) fish > $(DESTDIR)$(DATAROOTDIR)/fish/vendor_completions.d/$(PROG_PREFIX)$(prog).fish; \
+-	)
+ 
+ uninstall:
+ ifeq (${MULTICALL}, y)
+-- 
+2.25.1
+
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 1/2] package/uutils-coreutils: new package

[Thomas Petazzoni via buildroot]

On Fri, 17 Mar 2023 16:18:00 +0100
Sebastian Weyer <sebastian.weyer@smile.fr> wrote:

> This package is an implementation of coreutils written completely in rust.

Thanks for this patch. Unfortunately, it doesn't build. Tested on ARM,
it fails with:

The following warnings were emitted during compilation:

warning: arm-linux-g++.br_real: error: unrecognized command-line option ‘-m64’

error: failed to run custom build command for `uu_stdbuf_libstdbuf v0.0.21 (/home/thomas/projets/buildroot/output/build/uutils-coreutils-0.0.21/src/uu/stdbuf/src/libstdbuf)`

Caused by:
  process didn't exit successfully: `/home/thomas/projets/buildroot/output/build/uutils-coreutils-0.0.21/target/release/build/uu_stdbuf_libstdbuf-a77654d14689edab/build-script-build` (exit status: 1)

So, some C++ code gets built with ARM cross-compiler, but with a -m64
option that doesn't make sense on ARM.

Note: it failed with 0.0.17 which was used in your patch, so I updated
to 0.0.21 (as you can see in the log above), but it fails in a similar
way.

Also, your patch will have to be updated for 0.0.21, and I think your 2
patches can be combined into one by changing:

install: build manpages completions

to just:

install:

Could you have a look, and send an updated patch?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot