[Buildroot] [PATCH 1/6] core: export a variable with the host's bitness

[Buildroot] [PATCH 1/6] core: export a variable with the host's bitness

[Yann E. MORIN]

Some host packages (e.g. host-pseudo) wants to know the host bitness,
and they can fail to detect it in some corner cases (as reported by a
few users on IRC).

Export a variable with the bitness of the host, so that those packages
can be made to behave.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 Makefile | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Makefile b/Makefile
index 4ea64b3..78f72bc 100644
--- a/Makefile
+++ b/Makefile
@@ -340,6 +340,12 @@ export HOSTARCH := $(shell LC_ALL=C $(HOSTCC_NOCCACHE) -v 2>&1 | \
 	    -e 's/macppc/powerpc/' \
 	    -e 's/sh.*/sh/' )
 
+ifeq ($(filter %64,$(HOSTARCH)),)
+HOSTARCH_BITS := 32
+else
+HOSTARCH_BITS := 64
+endif
+
 HOSTCC_VERSION := $(shell $(HOSTCC_NOCCACHE) --version | \
 	sed -n -r 's/^.* ([0-9]*)\.([0-9]*)\.([0-9]*)[ ]*.*/\1 \2/p')
 
-- 
2.7.4

[Buildroot] [PATCH 2/6] package/pseudo: enforce the host bitness

[Yann E. MORIN]

pseudo can detect the host bitness, but is not sure about it: it checks
what type of file /bin/sh is, using file(1).

However, in some conditions, /bin/sh can be of a different bitness than
the rest of the system (weird, but not impossible), which causes build
issues.

Just enforce the bitness, so that pseudo needs not (wrongly) guess it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 package/pseudo/pseudo.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
index 67538a6..3b0ba86 100644
--- a/package/pseudo/pseudo.mk
+++ b/package/pseudo/pseudo.mk
@@ -17,6 +17,7 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
 # configure script is not generated by autoconf, so passing --libdir
 # is necessary, even if the infrastructure passes --prefix already.
 HOST_PSEUDO_CONF_OPTS = \
+	--bits=$(HOSTARCH_BITS) \
 	--libdir=$(HOST_DIR)/usr/lib \
 	--with-sqlite=$(HOST_DIR)/usr
 
-- 
2.7.4

[Buildroot] [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot

[Yann E. MORIN]

Hello All!

When we switched from using fakeroot to using pseudo, a lot of
assumptions were made, in that we assumed pseudo was by default
behaving as fakeroot did.

That is wrong. pseudo is a little bit more versatile than fakeroot
is, so using it is a little bit more involved.

First, its buildsystem is a bit convoluted, and can incorrectly guess
the host charateristics, like bitness.

Second, it needs to be told at runtime where to look at, and put its
own stuff. It correctly gueses, but prints a warning that it had to
gues, and the location to store its DB is not optimum (even if not
incorrect).

So, this series ensures that pseudo is correctly built and that we
correctly specify the runtime configuration.


Regards,
Yann E. MORIN.


The following changes since commit adf31efcd433712f546f2a3bcdb18abe650db27b

  board/boundarydevices: fix boot partition number in bootscript (2016-11-07 22:09:37 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to f7a170d2df33fe2825b401df89702f02e053c81f

  fs: call pseudo as instructed by the pseudo package (2016-11-07 22:22:24 +0100)


----------------------------------------------------------------
Yann E. MORIN (6):
      core: export a variable with the host's bitness
      package/pseudo: enforce the host bitness
      package/pseudo: force rpath and pass our host CFLAGS
      package/pseudo: expose the host binary in a variable
      package/pseudo: fix fakeroot wrapper to correctly use pseudo
      fs: call pseudo as instructed by the pseudo package

 Makefile                 |  6 ++++++
 fs/common.mk             |  2 +-
 package/pseudo/pseudo.mk | 13 ++++++++++++-
 3 files changed, 19 insertions(+), 2 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/6] package/pseudo: force rpath and pass our host CFLAGS

[Yann E. MORIN]

pseudo whines when those are not set, and tries to find a suitable
value; this is usually correct but risk an incorrect guess in corner
cases.

Rather than leaving those in guess-mode, just force them to values we
know are correct.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 package/pseudo/pseudo.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
index 3b0ba86..98c7de5 100644
--- a/package/pseudo/pseudo.mk
+++ b/package/pseudo/pseudo.mk
@@ -17,6 +17,8 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
 # configure script is not generated by autoconf, so passing --libdir
 # is necessary, even if the infrastructure passes --prefix already.
 HOST_PSEUDO_CONF_OPTS = \
+	--cflags='$(HOSTCFLAGS)' \
+	--with-rpath=$(HOST_DIR)/usr/lib \
 	--bits=$(HOSTARCH_BITS) \
 	--libdir=$(HOST_DIR)/usr/lib \
 	--with-sqlite=$(HOST_DIR)/usr
-- 
2.7.4

[Buildroot] [PATCH 4/6] package/pseudo: expose the host binary in a variable

[Yann E. MORIN]

Running pseudo is more involved than running fakeroot. In the transition
from using fakeroot, we just did not account for the extra requirements.

First, we explicitly tell pseudo where it is, otherwise it tries to
guess. Its guess is correct, but it prints a warning, which is not nice.

Second, we tell it where to find the passwd and group files in case it
has to emulate access to them. We currently do not use that feature, but
better safe than sorry.

Third, pseudo spawns a background daemon, and talks to it (when fakeroot
would emulate the state all in the current process' state, pseudo uses
the daemon to coordinate the state across multiple processes). We are
not much interested in the daemon lingering around, so we just tell it
to termintate as soon as the last clients quits (this can take up to one
second).

Fourth and last, pseudo always stores its internal database when
exiting, and reloads it when spawned. The database is by default stored
in a sub-directory of the prefix it was installed in, but this is
impractical for us, in case the host-dir is outside the config dir
$(BASE_DIR). We want the database to be specific to the one config dir
we are building. So, we store the database in a (hidden) sub-dir of the
build dir, thus ensuring it is never shared with another build.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 package/pseudo/pseudo.mk | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
index 98c7de5..2a3e31f 100644
--- a/package/pseudo/pseudo.mk
+++ b/package/pseudo/pseudo.mk
@@ -23,6 +23,13 @@ HOST_PSEUDO_CONF_OPTS = \
 	--libdir=$(HOST_DIR)/usr/lib \
 	--with-sqlite=$(HOST_DIR)/usr
 
+HOST_PSEUDO = \
+	PSEUDO_PREFIX=$(HOST_DIR)/usr \
+	PSEUDO_PASSWD=$(TARGET_DIR) \
+	PSEUDO_OPTS=-t0 \
+	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
+	$(HOST_DIR)/usr/bin/pseudo
+
 define HOST_PSEUDO_FAKEROOT_SYMLINK
 	ln -sf pseudo $(HOST_DIR)/usr/bin/fakeroot
 endef
-- 
2.7.4

[Buildroot] [PATCH 5/6] package/pseudo: fix fakeroot wrapper to correctly use pseudo

[Yann E. MORIN]

So far, we expected pseudo to behave like fakeroot would. This is not
correct: we need to export a few variables to make pseudo look like it
is behaving like fakeroot.

Provide a wrapper to impersonate fakeroot, instead of doing a symlink.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 package/pseudo/pseudo.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
index 2a3e31f..cc1bcdd 100644
--- a/package/pseudo/pseudo.mk
+++ b/package/pseudo/pseudo.mk
@@ -31,7 +31,8 @@ HOST_PSEUDO = \
 	$(HOST_DIR)/usr/bin/pseudo
 
 define HOST_PSEUDO_FAKEROOT_SYMLINK
-	ln -sf pseudo $(HOST_DIR)/usr/bin/fakeroot
+	printf '#!/bin/sh\n%s "$${@}"\n' '$(HOST_PSEUDO)' >$(HOST_DIR)/usr/bin/fakeroot
+	chmod 755 $(HOST_DIR)/usr/bin/fakeroot
 endef
 HOST_PSEUDO_POST_INSTALL_HOOKS += HOST_PSEUDO_FAKEROOT_SYMLINK
 
-- 
2.7.4

[Buildroot] [PATCH 6/6] fs: call pseudo as instructed by the pseudo package

[Yann E. MORIN]

Use the newly introduced HOST_PSEUDO variable to call pseudo, ensuring
the proper environment variables are set.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/common.mk b/fs/common.mk
index 2dbef4d..452b0cc 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -97,7 +97,7 @@ endif
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
-	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
+	PATH=$$(BR_PATH) $$(HOST_PSEUDO) -- $$(FAKEROOT_SCRIPT)
 	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
 	- at rm -f $$(FAKEROOT_SCRIPT) $$(FULL_DEVICE_TABLE)
 ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),)
-- 
2.7.4

[Buildroot] [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot

[Erico Nunes]

Hello,

On Mon, Nov 7, 2016 at 10:23 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Hello All!
>
> When we switched from using fakeroot to using pseudo, a lot of
> assumptions were made, in that we assumed pseudo was by default
> behaving as fakeroot did.
>
> That is wrong. pseudo is a little bit more versatile than fakeroot
> is, so using it is a little bit more involved.
>
> First, its buildsystem is a bit convoluted, and can incorrectly guess
> the host charateristics, like bitness.
>
> Second, it needs to be told at runtime where to look at, and put its
> own stuff. It correctly gueses, but prints a warning that it had to
> gues, and the location to store its DB is not optimum (even if not
> incorrect).
>
> So, this series ensures that pseudo is correctly built and that we
> correctly specify the runtime configuration.
>
>
> Regards,
> Yann E. MORIN.
>
>
> The following changes since commit adf31efcd433712f546f2a3bcdb18abe650db27b
>
>   board/boundarydevices: fix boot partition number in bootscript (2016-11-07 22:09:37 +0100)
>
>
> are available in the git repository at:
>
>   git://git.buildroot.org/~ymorin/git/buildroot.git
>
> for you to fetch changes up to f7a170d2df33fe2825b401df89702f02e053c81f
>
>   fs: call pseudo as instructed by the pseudo package (2016-11-07 22:22:24 +0100)
>
>
> ----------------------------------------------------------------
> Yann E. MORIN (6):
>       core: export a variable with the host's bitness
>       package/pseudo: enforce the host bitness
>       package/pseudo: force rpath and pass our host CFLAGS
>       package/pseudo: expose the host binary in a variable
>       package/pseudo: fix fakeroot wrapper to correctly use pseudo
>       fs: call pseudo as instructed by the pseudo package
>
>  Makefile                 |  6 ++++++
>  fs/common.mk             |  2 +-
>  package/pseudo/pseudo.mk | 13 ++++++++++++-
>  3 files changed, 19 insertions(+), 2 deletions(-)

I have tested this series on x86_64 and on an aarch64 build machine,
which was failing to build pseudo prior to this. Both running Fedora.
So, for the series:

Tested-by: Erico Nunes <nunes.erico@gmail.com>


Thanks for this fix.

Erico

[Buildroot] [PATCH 1/6] core: export a variable with the host's bitness

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> Some host packages (e.g. host-pseudo) wants to know the host bitness,
> and they can fail to detect it in some corner cases (as reported by a
> few users on IRC).
> 
> Export a variable with the bitness of the host, so that those packages
  ^^^^^^ You're not actually exporting it - fortunately.

> can be made to behave.

 I'm a bit hesitant to introduce a global infra variable like this (which makes
top-level Makefile longer - not the direction we want to go to) with just one
consumer, and where we hope the package can be fixed to not need this hack
anymore...

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> ---
>  Makefile | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 4ea64b3..78f72bc 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -340,6 +340,12 @@ export HOSTARCH := $(shell LC_ALL=C $(HOSTCC_NOCCACHE) -v 2>&1 | \
>  	    -e 's/macppc/powerpc/' \
>  	    -e 's/sh.*/sh/' )
>  
> +ifeq ($(filter %64,$(HOSTARCH)),)
> +HOSTARCH_BITS := 32

 There is no need to use := here, just use =

> +else
> +HOSTARCH_BITS := 64

 And here of course.

 Regards,
 Arnout

> +endif
> +
>  HOSTCC_VERSION := $(shell $(HOSTCC_NOCCACHE) --version | \
>  	sed -n -r 's/^.* ([0-9]*)\.([0-9]*)\.([0-9]*)[ ]*.*/\1 \2/p')
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 2/6] package/pseudo: enforce the host bitness

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> pseudo can detect the host bitness, but is not sure about it: it checks
> what type of file /bin/sh is, using file(1).
> 
> However, in some conditions, /bin/sh can be of a different bitness than
> the rest of the system (weird, but not impossible), which causes build
> issues.
> 
> Just enforce the bitness, so that pseudo needs not (wrongly) guess it.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> ---
>  package/pseudo/pseudo.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> index 67538a6..3b0ba86 100644
> --- a/package/pseudo/pseudo.mk
> +++ b/package/pseudo/pseudo.mk
> @@ -17,6 +17,7 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
>  # configure script is not generated by autoconf, so passing --libdir
>  # is necessary, even if the infrastructure passes --prefix already.
>  HOST_PSEUDO_CONF_OPTS = \
> +	--bits=$(HOSTARCH_BITS) \

 I would be just as happy with
	--bits=$(if $(filter %64,$(HOSTARCH)),32,64)


 Regards,
 Arnout

>  	--libdir=$(HOST_DIR)/usr/lib \
>  	--with-sqlite=$(HOST_DIR)/usr
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 3/6] package/pseudo: force rpath and pass our host CFLAGS

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> pseudo whines when those are not set, and tries to find a suitable
> value; this is usually correct but risk an incorrect guess in corner
> cases.
> 
> Rather than leaving those in guess-mode, just force them to values we
> know are correct.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  package/pseudo/pseudo.mk | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> index 3b0ba86..98c7de5 100644
> --- a/package/pseudo/pseudo.mk
> +++ b/package/pseudo/pseudo.mk
> @@ -17,6 +17,8 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
>  # configure script is not generated by autoconf, so passing --libdir
>  # is necessary, even if the infrastructure passes --prefix already.
>  HOST_PSEUDO_CONF_OPTS = \
> +	--cflags='$(HOSTCFLAGS)' \
> +	--with-rpath=$(HOST_DIR)/usr/lib \
>  	--bits=$(HOSTARCH_BITS) \
>  	--libdir=$(HOST_DIR)/usr/lib \
>  	--with-sqlite=$(HOST_DIR)/usr
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot

[Patrick Keroulas]

Hello all,

----- Original Message -----
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> To: buildroot at buildroot.org
> Cc: "Erico Nunes" <nunes.erico@gmail.com>, "Ga?l PORTAY" <gael.portay@savoirfairelinux.com>, "Julien BOIBESSOT"
> <julien.boibessot@free.fr>, "Patrick Keroulas" <patrick.keroulas@savoirfairelinux.com>, "Yann E. MORIN"
> <yann.morin.1998@free.fr>
> Sent: Monday, November 7, 2016 4:23:37 PM
> Subject: [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot

> Hello All!

> When we switched from using fakeroot to using pseudo, a lot of
> assumptions were made, in that we assumed pseudo was by default
> behaving as fakeroot did.

> That is wrong. pseudo is a little bit more versatile than fakeroot
> is, so using it is a little bit more involved.

> First, its buildsystem is a bit convoluted, and can incorrectly guess
> the host charateristics, like bitness.

> Second, it needs to be told at runtime where to look at, and put its
> own stuff. It correctly gueses, but prints a warning that it had to
> gues, and the location to store its DB is not optimum (even if not
> incorrect).

> So, this series ensures that pseudo is correctly built and that we
> correctly specify the runtime configuration.

I build my firmware inside a Docker container, in which a minimal 
Ubuntu x86-64 distro runs with no libc6-dev-i386 installed.
However, '-m32' was polluting the compiler options and the x86-64
headers couldn't be found. This patchset fixes the build.

Tested-by: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>

Thanks,

Patrick

[Buildroot] [PATCH 4/6] package/pseudo: expose the host binary in a variable

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> Running pseudo is more involved than running fakeroot. In the transition
> from using fakeroot, we just did not account for the extra requirements.
> 
> First, we explicitly tell pseudo where it is, otherwise it tries to
> guess. Its guess is correct, but it prints a warning, which is not nice.
> 
> Second, we tell it where to find the passwd and group files in case it
> has to emulate access to them. We currently do not use that feature, but
> better safe than sorry.
> 
> Third, pseudo spawns a background daemon, and talks to it (when fakeroot
> would emulate the state all in the current process' state, pseudo uses
> the daemon to coordinate the state across multiple processes). We are
> not much interested in the daemon lingering around, so we just tell it
> to termintate as soon as the last clients quits (this can take up to one
> second).
> 
> Fourth and last, pseudo always stores its internal database when
> exiting, and reloads it when spawned. The database is by default stored
> in a sub-directory of the prefix it was installed in, but this is
> impractical for us, in case the host-dir is outside the config dir
> $(BASE_DIR). We want the database to be specific to the one config dir

 $(HOST_DIR), even if outside of $(BASE_DIR), must also be specific to one
config dir. Remember that $(STAGING_DIR) resides under $(HOST_DIR). So this
argument is not really valid. However, I do fully agree that
$(HOST_DIR)/usr/var/pseudo is *not* the best place for it, and that your
proposed location is better.

> we are building. So, we store the database in a (hidden) sub-dir of the
> build dir, thus ensuring it is never shared with another build.

 I would like to add the following:

We pass these options as environment variables in the pseudo invocation,
nicely wrapped in the HOST_PSEUDO variable. An alternative would be to
create a pseudo wrapper script, but that is less attractive for the
following reasons.

- We don't expect users to need to call pseudo from post-build/image
  scripts, because we already support BR2_ROOTFS_POST_FAKEROOT_SCRIPT.

- Creating a script is a bit more complicated, e.g. with relocatable
  host directory in mind.


> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

... but I like my bikeshed to be coloured differently...

> ---
>  package/pseudo/pseudo.mk | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> index 98c7de5..2a3e31f 100644
> --- a/package/pseudo/pseudo.mk
> +++ b/package/pseudo/pseudo.mk
> @@ -23,6 +23,13 @@ HOST_PSEUDO_CONF_OPTS = \
>  	--libdir=$(HOST_DIR)/usr/lib \
>  	--with-sqlite=$(HOST_DIR)/usr
>  
> +HOST_PSEUDO = \

 We use $(MAKE), $(TAR), $(QT_QMAKE), $(QT5_QMAKE), $(BR2_CMAKE), $(LUA_RUN),
$(PERL_RUN). Nowhere do we use $(HOST_FOO) to define a command. I admit that all
of these are not completely consistent, but I think the _RUN pattern fits best
for this situation, so how about PSEUDO_RUN?

> +	PSEUDO_PREFIX=$(HOST_DIR)/usr \
> +	PSEUDO_PASSWD=$(TARGET_DIR) \
> +	PSEUDO_OPTS=-t0 \
> +	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \

 I don't really like hidden files/directories in BUILD_DIR. I was thinking more
along the lines of $(BASE_DIR)/target-pseudodb. But I'm OK with your proposal as
well.

> +	$(HOST_DIR)/usr/bin/pseudo

 Perhaps it's better to provide the options on the command line, as far as
possible. So
	PSEUDO_PASSWD=$(TARGET_DIR) \
	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
	$(HOST_DIR)/usr/bin/pseudo -t0 -P $(HOST_DIR)/usr


 Regards,
 Arnout

> +
>  define HOST_PSEUDO_FAKEROOT_SYMLINK
>  	ln -sf pseudo $(HOST_DIR)/usr/bin/fakeroot
>  endef
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 5/6] package/pseudo: fix fakeroot wrapper to correctly use pseudo

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> So far, we expected pseudo to behave like fakeroot would. This is not
> correct: we need to export a few variables to make pseudo look like it
> is behaving like fakeroot.
> 
> Provide a wrapper to impersonate fakeroot, instead of doing a symlink.

 Not nice, because it introduces absolute paths in host dir which is not good
for relocation.

 I'm thinking we shouldn't have introduced this symlink to begin with. I did
suggest it on IRC, but I wasn't thinking. fakeroot could never be called by
post-image scripts anyway, because the fakeroot database isn't saved, whatever
you do under fakeroot is lost.

 Hence I'm more inclined to revert 7158403805f. If someone does need it, they
can get patches from the mailing list :-)

 Regards,
 Arnout

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> ---
>  package/pseudo/pseudo.mk | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> index 2a3e31f..cc1bcdd 100644
> --- a/package/pseudo/pseudo.mk
> +++ b/package/pseudo/pseudo.mk
> @@ -31,7 +31,8 @@ HOST_PSEUDO = \
>  	$(HOST_DIR)/usr/bin/pseudo
>  
>  define HOST_PSEUDO_FAKEROOT_SYMLINK
> -	ln -sf pseudo $(HOST_DIR)/usr/bin/fakeroot
> +	printf '#!/bin/sh\n%s "$${@}"\n' '$(HOST_PSEUDO)' >$(HOST_DIR)/usr/bin/fakeroot
> +	chmod 755 $(HOST_DIR)/usr/bin/fakeroot
>  endef
>  HOST_PSEUDO_POST_INSTALL_HOOKS += HOST_PSEUDO_FAKEROOT_SYMLINK
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 6/6] fs: call pseudo as instructed by the pseudo package

[Arnout Vandecappelle]

On 07-11-16 22:23, Yann E. MORIN wrote:
> Use the newly introduced HOST_PSEUDO variable to call pseudo, ensuring
> the proper environment variables are set.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> Cc: Erico Nunes <nunes.erico@gmail.com>
> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> ---
>  fs/common.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/common.mk b/fs/common.mk
> index 2dbef4d..452b0cc 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -97,7 +97,7 @@ endif
>  		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
>  	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
>  	chmod a+x $$(FAKEROOT_SCRIPT)
> -	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
> +	PATH=$$(BR_PATH) $$(HOST_PSEUDO) -- $$(FAKEROOT_SCRIPT)
>  	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
>  	- at rm -f $$(FAKEROOT_SCRIPT) $$(FULL_DEVICE_TABLE)
>  ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot

[Gaël PORTAY]

Hello all,

On Mon, Nov 07, 2016 at 05:26:55PM -0500, Patrick Keroulas wrote:
> Hello all,
> 
> ----- Original Message -----
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > To: buildroot at buildroot.org
> > Cc: "Erico Nunes" <nunes.erico@gmail.com>, "Ga?l PORTAY" <gael.portay@savoirfairelinux.com>, "Julien BOIBESSOT"
> > <julien.boibessot@free.fr>, "Patrick Keroulas" <patrick.keroulas@savoirfairelinux.com>, "Yann E. MORIN"
> > <yann.morin.1998@free.fr>
> > Sent: Monday, November 7, 2016 4:23:37 PM
> > Subject: [PATCH 0/6] package/pseudo: fix build and better mimic fakeroot
> 
> > Hello All!
> 
> > When we switched from using fakeroot to using pseudo, a lot of
> > assumptions were made, in that we assumed pseudo was by default
> > behaving as fakeroot did.
> 
> > That is wrong. pseudo is a little bit more versatile than fakeroot
> > is, so using it is a little bit more involved.
> 
> > First, its buildsystem is a bit convoluted, and can incorrectly guess
> > the host charateristics, like bitness.
> 
> > Second, it needs to be told at runtime where to look at, and put its
> > own stuff. It correctly gueses, but prints a warning that it had to
> > gues, and the location to store its DB is not optimum (even if not
> > incorrect).
> 
> > So, this series ensures that pseudo is correctly built and that we
> > correctly specify the runtime configuration.
> 
> I build my firmware inside a Docker container, in which a minimal 
> Ubuntu x86-64 distro runs with no libc6-dev-i386 installed.
> However, '-m32' was polluting the compiler options and the x86-64
> headers couldn't be found. This patchset fixes the build.
> 
> Tested-by: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> 
> Thanks,
> 
> Patrick
> 

Okay works fine on a Ubuntu 16.04 i686 via Vagrant!

$ vagrand ssh
ubuntu at ubuntu-xenial:~$ uname -a
Linux ubuntu-xenial 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:05 UTC 2016 i686 i686 i686 GNU/
ubuntu at ubuntu-xenial:~$ cd Workspaces/buildroot/ && make host-pseudo
(... full log at http://pastebin.ca/3737232)
cc  -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE  -DPSEUDO_PREFIX='"/home/ubuntu/Workspaces/buildroot/output/host/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.8.1"' -DUSE_MEMORY_DB  -DPSEUDO_PASSWD_FALLBACK='""'   -DPSEUDO_XATTR_SUPPORT       -O2 -g -L/home/ubuntu/Workspaces/buildroot/output/host/usr/lib -I/home/ubuntu/Workspaces/buildroot/output/host/usr/include -Wl,-R/home/ubuntu/Workspaces/buildroot/output/host/usr/lib -shared -o lib/libpseudo.so \
	pseudo_client.o pseudo_ipc.o \
	pseudo_wrappers.o pseudo_tables.o pseudo_util.o  -ldl -lpthread
(...)

You have my
Tested-by: Ga?l PORTAY <gael.portay@savoirfairelinux.com>

Thanks,
Ga?l

[Buildroot] [PATCH 5/6] package/pseudo: fix fakeroot wrapper to correctly use pseudo

[Yann E. MORIN]

Arnout, All,

On 2016-11-08 00:10 +0100, Arnout Vandecappelle spake thusly:
> 
> 
> On 07-11-16 22:23, Yann E. MORIN wrote:
> > So far, we expected pseudo to behave like fakeroot would. This is not
> > correct: we need to export a few variables to make pseudo look like it
> > is behaving like fakeroot.
> > 
> > Provide a wrapper to impersonate fakeroot, instead of doing a symlink.
> 
>  Not nice, because it introduces absolute paths in host dir which is not good
> for relocation.

Agreed. But note that we already have pkg-config as a wrapper to
pkgconf, and that one is not relocatable yet, as far as I can see...

I can rework this to make it more relocatable, indeed. I at least wanted
this series out for the core fixes about pseudo, so affected parties
could test the overall idea of the fixes.

>  I'm thinking we shouldn't have introduced this symlink to begin with. I did
> suggest it on IRC, but I wasn't thinking. fakeroot could never be called by
> post-image scripts anyway, because the fakeroot database isn't saved, whatever
> you do under fakeroot is lost.

Wrong.

If Buildroot generates a tarball, then all that needed root access is in
the tarball. So a post-image script that runs under fakeroot (even
*another* fakeroot instance) can safely extract that tarball and get all
the root-requiring settings and reasemble another that in one or more
filesystem images, as those would be emulated by fakeroot (as long as
the script does not exit the fakeroot instance in-between, that is).

That is something I've been using on a daily basis for years now...

Now, we could argue that such post-image scripts should be changed to
use pseudo instead. We could advertise that by providing this fakeroot
wrapper:

    #!/bin/sh
    echo "No, no, use pseudo now..." >&2
    exit 1

However, what we don't want is the user to use his host-provided
fakeroot, because that one might not be fit (e.g. lacking support for
xattrs when pseudo had stored them in the tarball, for example).

So, I still think we should either provide a catcher fakeroot as above,
or we provide a real wrapper. I like the catcher, above.

Regards,
Yann E. MORIN.

>  Hence I'm more inclined to revert 7158403805f. If someone does need it, they
> can get patches from the mailing list :-)
> 
>  Regards,
>  Arnout
> 
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> > Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> > Cc: Erico Nunes <nunes.erico@gmail.com>
> > Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> > ---
> >  package/pseudo/pseudo.mk | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> > index 2a3e31f..cc1bcdd 100644
> > --- a/package/pseudo/pseudo.mk
> > +++ b/package/pseudo/pseudo.mk
> > @@ -31,7 +31,8 @@ HOST_PSEUDO = \
> >  	$(HOST_DIR)/usr/bin/pseudo
> >  
> >  define HOST_PSEUDO_FAKEROOT_SYMLINK
> > -	ln -sf pseudo $(HOST_DIR)/usr/bin/fakeroot
> > +	printf '#!/bin/sh\n%s "$${@}"\n' '$(HOST_PSEUDO)' >$(HOST_DIR)/usr/bin/fakeroot
> > +	chmod 755 $(HOST_DIR)/usr/bin/fakeroot
> >  endef
> >  HOST_PSEUDO_POST_INSTALL_HOOKS += HOST_PSEUDO_FAKEROOT_SYMLINK
> >  
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/6] core: export a variable with the host's bitness

[Yann E. MORIN]

Arnout, All,

On 2016-11-07 23:15 +0100, Arnout Vandecappelle spake thusly:
> On 07-11-16 22:23, Yann E. MORIN wrote:
> > Some host packages (e.g. host-pseudo) wants to know the host bitness,
> > and they can fail to detect it in some corner cases (as reported by a
> > few users on IRC).
> > 
> > Export a variable with the bitness of the host, so that those packages
>   ^^^^^^ You're not actually exporting it - fortunately.

Unfortunate choice of wording. Indeed, we need not 'export' it. We just
need to make it available.

Arguably, HOSTARCH itself should not be exported. But we need it in
Kconfig, we could add it to COMMON_CONFIG_ENV instead.

Note that gnu-efi is using HOSTARCH )package/gnu-efi/0001-*.patch), so
exporting our own may interfere with its buildsystem (to be confirmed).

> > can be made to behave.
> 
>  I'm a bit hesitant to introduce a global infra variable like this (which makes
> top-level Makefile longer - not the direction we want to go to) with just one
> consumer, and where we hope the package can be fixed to not need this hack
> anymore...

I will drop this patch and only use that in pseduo.
Thanks! :-)

Regards,
Yann E. MORIN.

> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> > Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> > Cc: Erico Nunes <nunes.erico@gmail.com>
> > Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> > ---
> >  Makefile | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/Makefile b/Makefile
> > index 4ea64b3..78f72bc 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -340,6 +340,12 @@ export HOSTARCH := $(shell LC_ALL=C $(HOSTCC_NOCCACHE) -v 2>&1 | \
> >  	    -e 's/macppc/powerpc/' \
> >  	    -e 's/sh.*/sh/' )
> >  
> > +ifeq ($(filter %64,$(HOSTARCH)),)
> > +HOSTARCH_BITS := 32
> 
>  There is no need to use := here, just use =
> 
> > +else
> > +HOSTARCH_BITS := 64
> 
>  And here of course.
> 
>  Regards,
>  Arnout
> 
> > +endif
> > +
> >  HOSTCC_VERSION := $(shell $(HOSTCC_NOCCACHE) --version | \
> >  	sed -n -r 's/^.* ([0-9]*)\.([0-9]*)\.([0-9]*)[ ]*.*/\1 \2/p')
> >  
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/6] package/pseudo: enforce the host bitness

[Yann E. MORIN]

Arnout, All,

On 2016-11-07 23:16 +0100, Arnout Vandecappelle spake thusly:
> 
> 
> On 07-11-16 22:23, Yann E. MORIN wrote:
> > pseudo can detect the host bitness, but is not sure about it: it checks
> > what type of file /bin/sh is, using file(1).
> > 
> > However, in some conditions, /bin/sh can be of a different bitness than
> > the rest of the system (weird, but not impossible), which causes build
> > issues.
> > 
> > Just enforce the bitness, so that pseudo needs not (wrongly) guess it.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> > Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> > Cc: Erico Nunes <nunes.erico@gmail.com>
> > Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> > ---
> >  package/pseudo/pseudo.mk | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> > index 67538a6..3b0ba86 100644
> > --- a/package/pseudo/pseudo.mk
> > +++ b/package/pseudo/pseudo.mk
> > @@ -17,6 +17,7 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
> >  # configure script is not generated by autoconf, so passing --libdir
> >  # is necessary, even if the infrastructure passes --prefix already.
> >  HOST_PSEUDO_CONF_OPTS = \
> > +	--bits=$(HOSTARCH_BITS) \
> 
>  I would be just as happy with
> 	--bits=$(if $(filter %64,$(HOSTARCH)),32,64)

Yep, that's nice. Except it's reversed, isn't it? I.e. if we find '64'
in HOSTARCH, then we want to use '64', not '32', right? So:

    --bits=$(if $(filter %64,$(HOSTARCH)),64,32)

(Unless I missed something...)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 4/6] package/pseudo: expose the host binary in a variable

[Yann E. MORIN]

Arnout, All,

On 2016-11-08 00:01 +0100, Arnout Vandecappelle spake thusly:
> On 07-11-16 22:23, Yann E. MORIN wrote:
> > Running pseudo is more involved than running fakeroot. In the transition
> > from using fakeroot, we just did not account for the extra requirements.
> > 
> > First, we explicitly tell pseudo where it is, otherwise it tries to
> > guess. Its guess is correct, but it prints a warning, which is not nice.
> > 
> > Second, we tell it where to find the passwd and group files in case it
> > has to emulate access to them. We currently do not use that feature, but
> > better safe than sorry.
> > 
> > Third, pseudo spawns a background daemon, and talks to it (when fakeroot
> > would emulate the state all in the current process' state, pseudo uses
> > the daemon to coordinate the state across multiple processes). We are
> > not much interested in the daemon lingering around, so we just tell it
> > to termintate as soon as the last clients quits (this can take up to one
> > second).
> > 
> > Fourth and last, pseudo always stores its internal database when
> > exiting, and reloads it when spawned. The database is by default stored
> > in a sub-directory of the prefix it was installed in, but this is
> > impractical for us, in case the host-dir is outside the config dir
> > $(BASE_DIR). We want the database to be specific to the one config dir
> 
>  $(HOST_DIR), even if outside of $(BASE_DIR), must also be specific to one
> config dir. Remember that $(STAGING_DIR) resides under $(HOST_DIR). So this
> argument is not really valid. However, I do fully agree that
> $(HOST_DIR)/usr/var/pseudo is *not* the best place for it, and that your
> proposed location is better.

OK, I'll rephrase accordingly.

> > we are building. So, we store the database in a (hidden) sub-dir of the
> > build dir, thus ensuring it is never shared with another build.
> 
>  I would like to add the following:
> 
> We pass these options as environment variables in the pseudo invocation,
> nicely wrapped in the HOST_PSEUDO variable. An alternative would be to
> create a pseudo wrapper script, but that is less attractive for the
> following reasons.
> 
> - We don't expect users to need to call pseudo from post-build/image
>   scripts, because we already support BR2_ROOTFS_POST_FAKEROOT_SCRIPT.

Wrong for post-image, as I already explained.

> - Creating a script is a bit more complicated, e.g. with relocatable
>   host directory in mind.

So, a pseudo wrapper is iundeed better, because new post-image scrips
could directly use pseudo instead of fakeroot, and the fakeroot wrapper
could be a symlink to the pseudo wrapper now.

> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
> > Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
> > Cc: Erico Nunes <nunes.erico@gmail.com>
> > Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> ... but I like my bikeshed to be coloured differently...

:-)

> > ---
> >  package/pseudo/pseudo.mk | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
> > index 98c7de5..2a3e31f 100644
> > --- a/package/pseudo/pseudo.mk
> > +++ b/package/pseudo/pseudo.mk
> > @@ -23,6 +23,13 @@ HOST_PSEUDO_CONF_OPTS = \
> >  	--libdir=$(HOST_DIR)/usr/lib \
> >  	--with-sqlite=$(HOST_DIR)/usr
> >  
> > +HOST_PSEUDO = \
> 
>  We use $(MAKE), $(TAR), $(QT_QMAKE), $(QT5_QMAKE), $(BR2_CMAKE), $(LUA_RUN),
> $(PERL_RUN). Nowhere do we use $(HOST_FOO) to define a command. I admit that all
> of these are not completely consistent, but I think the _RUN pattern fits best
> for this situation, so how about PSEUDO_RUN?

And we have PKG_CONFIG_HOST_BINARY... ;-)

I don't much care about hte name, but the _RUN scheme is not really
appealing to me. Consistency is nice, though, but we can't really say
that _RUN is consistent, sionce we only have to uses now, and so many
different variants...

Let's see what the powers-that-be prefer! ;-)

> > +	PSEUDO_PREFIX=$(HOST_DIR)/usr \
> > +	PSEUDO_PASSWD=$(TARGET_DIR) \
> > +	PSEUDO_OPTS=-t0 \
> > +	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
> 
>  I don't really like hidden files/directories in BUILD_DIR. I was thinking more
> along the lines of $(BASE_DIR)/target-pseudodb. But I'm OK with your proposal as
> well.

Well, I don;t really like to expose to the user our internals. The DB is
internal state of Buildroot. A dot-directory is nice for that...

> > +	$(HOST_DIR)/usr/bin/pseudo
> 
>  Perhaps it's better to provide the options on the command line, as far as
> possible. So
> 	PSEUDO_PASSWD=$(TARGET_DIR) \
> 	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
> 	$(HOST_DIR)/usr/bin/pseudo -t0 -P $(HOST_DIR)/usr

And we need not provide them at all if we provide a wrapper...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/6] package/pseudo: enforce the host bitness

[Arnout Vandecappelle]

On 08-11-16 18:29, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2016-11-07 23:16 +0100, Arnout Vandecappelle spake thusly:
>>
>>
>> On 07-11-16 22:23, Yann E. MORIN wrote:
>>> pseudo can detect the host bitness, but is not sure about it: it checks
>>> what type of file /bin/sh is, using file(1).
>>>
>>> However, in some conditions, /bin/sh can be of a different bitness than
>>> the rest of the system (weird, but not impossible), which causes build
>>> issues.
>>>
>>> Just enforce the bitness, so that pseudo needs not (wrongly) guess it.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
>>> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
>>> Cc: Erico Nunes <nunes.erico@gmail.com>
>>> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
>>> ---
>>>  package/pseudo/pseudo.mk | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
>>> index 67538a6..3b0ba86 100644
>>> --- a/package/pseudo/pseudo.mk
>>> +++ b/package/pseudo/pseudo.mk
>>> @@ -17,6 +17,7 @@ HOST_PSEUDO_DEPENDENCIES = host-attr host-sqlite
>>>  # configure script is not generated by autoconf, so passing --libdir
>>>  # is necessary, even if the infrastructure passes --prefix already.
>>>  HOST_PSEUDO_CONF_OPTS = \
>>> +	--bits=$(HOSTARCH_BITS) \
>>
>>  I would be just as happy with
>> 	--bits=$(if $(filter %64,$(HOSTARCH)),32,64)
> 
> Yep, that's nice. Except it's reversed, isn't it? I.e. if we find '64'
> in HOSTARCH, then we want to use '64', not '32', right? So:
> 
>     --bits=$(if $(filter %64,$(HOSTARCH)),64,32)

 Ugh, yes, of course, I had filter-out in my head for some reason. Ah, no, it's
because the condition in patch 1/6 was reversed that I was confused.

 Regards,
 Arnout

> 
> (Unless I missed something...)
> 
> Regards,
> Yann E. MORIN.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 4/6] package/pseudo: expose the host binary in a variable

[Arnout Vandecappelle]

On 08-11-16 18:36, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2016-11-08 00:01 +0100, Arnout Vandecappelle spake thusly:
>> On 07-11-16 22:23, Yann E. MORIN wrote:
>>> Running pseudo is more involved than running fakeroot. In the transition
>>> from using fakeroot, we just did not account for the extra requirements.
>>>
>>> First, we explicitly tell pseudo where it is, otherwise it tries to
>>> guess. Its guess is correct, but it prints a warning, which is not nice.
>>>
>>> Second, we tell it where to find the passwd and group files in case it
>>> has to emulate access to them. We currently do not use that feature, but
>>> better safe than sorry.
>>>
>>> Third, pseudo spawns a background daemon, and talks to it (when fakeroot
>>> would emulate the state all in the current process' state, pseudo uses
>>> the daemon to coordinate the state across multiple processes). We are
>>> not much interested in the daemon lingering around, so we just tell it
>>> to termintate as soon as the last clients quits (this can take up to one
>>> second).
>>>
>>> Fourth and last, pseudo always stores its internal database when
>>> exiting, and reloads it when spawned. The database is by default stored
>>> in a sub-directory of the prefix it was installed in, but this is
>>> impractical for us, in case the host-dir is outside the config dir
>>> $(BASE_DIR). We want the database to be specific to the one config dir
>>
>>  $(HOST_DIR), even if outside of $(BASE_DIR), must also be specific to one
>> config dir. Remember that $(STAGING_DIR) resides under $(HOST_DIR). So this
>> argument is not really valid. However, I do fully agree that
>> $(HOST_DIR)/usr/var/pseudo is *not* the best place for it, and that your
>> proposed location is better.
> 
> OK, I'll rephrase accordingly.
> 
>>> we are building. So, we store the database in a (hidden) sub-dir of the
>>> build dir, thus ensuring it is never shared with another build.
>>
>>  I would like to add the following:
>>
>> We pass these options as environment variables in the pseudo invocation,
>> nicely wrapped in the HOST_PSEUDO variable. An alternative would be to
>> create a pseudo wrapper script, but that is less attractive for the
>> following reasons.
>>
>> - We don't expect users to need to call pseudo from post-build/image
>>   scripts, because we already support BR2_ROOTFS_POST_FAKEROOT_SCRIPT.
> 
> Wrong for post-image, as I already explained.
> 
>> - Creating a script is a bit more complicated, e.g. with relocatable
>>   host directory in mind.
> 
> So, a pseudo wrapper is iundeed better, because new post-image scrips
> could directly use pseudo instead of fakeroot, and the fakeroot wrapper
> could be a symlink to the pseudo wrapper now.

 In that case, why don't we call the wrapper 'fakeroot' and leave the pseudo
binary alone? In the assumption that people who call the pseudo binary directly
would know what they are doing so they'll add the right options, and people who
don't know what they're doing will call the fakeroot wrapper.

 Hm, no, on second thought, I don't particularly like that either. But I also
don't like "overwriting" pseudo with a wrapper. So then I propose to call the
wrapper 'pseudo-wrapper' and symlink fakeroot to it.


> 
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Ga?l PORTAY <gael.portay@savoirfairelinux.com>
>>> Cc: Patrick Keroulas <patrick.keroulas@savoirfairelinux.com>
>>> Cc: Erico Nunes <nunes.erico@gmail.com>
>>> Cc: Julien BOIBESSOT <julien.boibessot@free.fr>
>>
>> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>>
>> ... but I like my bikeshed to be coloured differently...
> 
> :-)
> 
>>> ---
>>>  package/pseudo/pseudo.mk | 7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/package/pseudo/pseudo.mk b/package/pseudo/pseudo.mk
>>> index 98c7de5..2a3e31f 100644
>>> --- a/package/pseudo/pseudo.mk
>>> +++ b/package/pseudo/pseudo.mk
>>> @@ -23,6 +23,13 @@ HOST_PSEUDO_CONF_OPTS = \
>>>  	--libdir=$(HOST_DIR)/usr/lib \
>>>  	--with-sqlite=$(HOST_DIR)/usr
>>>  
>>> +HOST_PSEUDO = \
>>
>>  We use $(MAKE), $(TAR), $(QT_QMAKE), $(QT5_QMAKE), $(BR2_CMAKE), $(LUA_RUN),
>> $(PERL_RUN). Nowhere do we use $(HOST_FOO) to define a command. I admit that all
>> of these are not completely consistent, but I think the _RUN pattern fits best
>> for this situation, so how about PSEUDO_RUN?
> 
> And we have PKG_CONFIG_HOST_BINARY... ;-)
> 
> I don't much care about hte name, but the _RUN scheme is not really
> appealing to me. Consistency is nice, though, but we can't really say
> that _RUN is consistent, sionce we only have to uses now, and so many
> different variants...
> 
> Let's see what the powers-that-be prefer! ;-)

 Well, if it's a wrapper script after all, then there is no need for a variable.
We don't have one for python either.

> 
>>> +	PSEUDO_PREFIX=$(HOST_DIR)/usr \
>>> +	PSEUDO_PASSWD=$(TARGET_DIR) \
>>> +	PSEUDO_OPTS=-t0 \
>>> +	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
>>
>>  I don't really like hidden files/directories in BUILD_DIR. I was thinking more
>> along the lines of $(BASE_DIR)/target-pseudodb. But I'm OK with your proposal as
>> well.
> 
> Well, I don;t really like to expose to the user our internals. The DB is
> internal state of Buildroot. A dot-directory is nice for that...

 Good point.


 Regards,
 Arnout

>>> +	$(HOST_DIR)/usr/bin/pseudo
>>
>>  Perhaps it's better to provide the options on the command line, as far as
>> possible. So
>> 	PSEUDO_PASSWD=$(TARGET_DIR) \
>> 	PSEUDO_LOCALSTATEDIR=$(BUILD_DIR)/.pseudodb \
>> 	$(HOST_DIR)/usr/bin/pseudo -t0 -P $(HOST_DIR)/usr
> 
> And we need not provide them at all if we provide a wrapper...
> 
> Regards,
> Yann E. MORIN.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF