Re: Bug#953537: xfsdump fails to install in /usr merged system.

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[peter green]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

(original message was sent to nathans@redhat.com 953537@bugs.debian.org and linux-xfs@vger.kernel.org re-sending as plain-text only to linux-xfs@vger.kernel.org)

This bug has now caused xfsdump to be kicked out of testing which is making amanda unbuildable in testing.


> Yes, what's really needed here is for a change to be merged upstream
> (as all other deb packaging artifacts are) otherwise this will keep
> getting lost in time.
To make it easier to upstream this I whipped up a patch that should solve the issue while only modifying the debian packaging and not touching the upstream makefiles. It is attached to this message and if I get no response I will likely do some further testing and then NMU it in Debian.

One issue I noticed is it's not all all obvious who upstream is. The sgi website listed in README seems to be long dead and there are no obvious upstream results in a google search for xfsdump. Gentoos page on xfsdump links to https://xfs.wiki.kernel.org but that page makes no mention of xfsdump.

I eventually poked around on git.kernel.org and my best guess is that https://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git/ is the upstream git repository and linux-xfs@vger.kernel.org is the appropriate mailing list, I would appreciate comments on whether or not this is correct and updates to the documentation to reflect whatever the correct location is.

[-- Attachment #2: xfsdump.debdiff --]
[-- Type: text/plain, Size: 2207 bytes --]

diff -Nru xfsdump-3.1.9/debian/changelog xfsdump-3.1.9+nmu1/debian/changelog
--- xfsdump-3.1.9/debian/changelog	2020-01-31 17:30:58.000000000 +0000
+++ xfsdump-3.1.9+nmu1/debian/changelog	2020-06-19 01:01:18.000000000 +0000
@@ -1,3 +1,13 @@
+xfsdump (3.1.9+nmu1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Create and remove symlinks in postinst/preinst rather than including them
+    in the package to support merged user systems. Based on a patch from
+    Goffredo Baroncelli but adjusted to avoid the need for modifying upstream
+    non-debian files. ( Closes: 953537 )
+
+ -- Peter Michael Green <plugwash@debian.org>  Fri, 19 Jun 2020 01:01:18 +0000
+
 xfsdump (3.1.9) unstable; urgency=low
 
   * New upstream release
diff -Nru xfsdump-3.1.9/debian/rules xfsdump-3.1.9+nmu1/debian/rules
--- xfsdump-3.1.9/debian/rules	2020-01-31 17:30:58.000000000 +0000
+++ xfsdump-3.1.9+nmu1/debian/rules	2020-06-19 01:01:18.000000000 +0000
@@ -44,6 +44,9 @@
 	-rm -rf $(dirme)
 	$(pkgme) $(MAKE) -C . install
 	$(pkgme) $(MAKE) dist
+	#remove the symlinks in /usr/sbin, the postinst will create them
+	#if appropriate for the users system 
+	rm -f debian/xfsdump/usr/sbin/xfsdump debian/xfsdump/usr/sbin/xfsrestore
 	dh_installdocs
 	dh_installchangelogs
 	dh_strip
diff -Nru xfsdump-3.1.9/debian/xfsdump.postinst xfsdump-3.1.9+nmu1/debian/xfsdump.postinst
--- xfsdump-3.1.9/debian/xfsdump.postinst	1970-01-01 00:00:00.000000000 +0000
+++ xfsdump-3.1.9+nmu1/debian/xfsdump.postinst	2020-06-19 00:59:32.000000000 +0000
@@ -0,0 +1,12 @@
+#!/bin/sh
+set -e
+
+if [ "$1" = 'configure' ]; then
+  for file in xfsdump xfsrestore; do
+    if [ ! -e /usr/sbin/$file ]; then
+      ln -s /sbin/$file /usr/sbin/$file
+    fi
+  done
+fi
+
+#DEBHELPER#
diff -Nru xfsdump-3.1.9/debian/xfsdump.preinst xfsdump-3.1.9+nmu1/debian/xfsdump.preinst
--- xfsdump-3.1.9/debian/xfsdump.preinst	1970-01-01 00:00:00.000000000 +0000
+++ xfsdump-3.1.9+nmu1/debian/xfsdump.preinst	2020-06-19 01:01:18.000000000 +0000
@@ -0,0 +1,12 @@
+#!/bin/sh
+set -e
+
+if [ "$1" = 'remove' ]; then
+  for file in xfsdump xfsrestore; do
+    if [ -L /usr/sbin/$file ]; then
+      rm /usr/sbin/$file
+    fi
+  done
+fi
+
+#DEBHELPER#

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[Darrick J. Wong]

On Fri, Jun 19, 2020 at 05:05:00AM +0100, peter green wrote:
> (original message was sent to nathans@redhat.com
> 953537@bugs.debian.org and linux-xfs@vger.kernel.org re-sending as
> plain-text only to linux-xfs@vger.kernel.org)
> 
> This bug has now caused xfsdump to be kicked out of testing which is
> making amanda unbuildable in testing.

Uhoh...

> 
> 
> > Yes, what's really needed here is for a change to be merged upstream
> > (as all other deb packaging artifacts are) otherwise this will keep
> > getting lost in time.
> To make it easier to upstream this I whipped up a patch that should
> solve the issue while only modifying the debian packaging and not
> touching the upstream makefiles. It is attached to this message and if
> I get no response I will likely do some further testing and then NMU
> it in Debian.
> 
> One issue I noticed is it's not all all obvious who upstream is. The
> sgi website listed in README seems to be long dead and there are no
> obvious upstream results in a google search for xfsdump. Gentoos page
> on xfsdump links to https://xfs.wiki.kernel.org but that page makes no
> mention of xfsdump.
> 
> I eventually poked around on git.kernel.org and my best guess is that
> https://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git/ is the upstream
> git repository and linux-xfs@vger.kernel.org is the appropriate
> mailing list, I would appreciate comments on whether or not this is
> correct and updates to the documentation to reflect whatever the
> correct location is.

Yep, you've found us. :)

Uh... seeing how /sbin seems to be a symlink to /usr/sbin on more and
more distros now, how about we just change the upstream makefile to dump
them in /usr/sbin and forget all about the symlinks?

(He says, wondering what the actual maintainer will say...)

--D

> diff -Nru xfsdump-3.1.9/debian/changelog xfsdump-3.1.9+nmu1/debian/changelog
> --- xfsdump-3.1.9/debian/changelog	2020-01-31 17:30:58.000000000 +0000
> +++ xfsdump-3.1.9+nmu1/debian/changelog	2020-06-19 01:01:18.000000000 +0000
> @@ -1,3 +1,13 @@
> +xfsdump (3.1.9+nmu1) UNRELEASED; urgency=medium
> +
> +  * Non-maintainer upload.
> +  * Create and remove symlinks in postinst/preinst rather than including them
> +    in the package to support merged user systems. Based on a patch from
> +    Goffredo Baroncelli but adjusted to avoid the need for modifying upstream
> +    non-debian files. ( Closes: 953537 )
> +
> + -- Peter Michael Green <plugwash@debian.org>  Fri, 19 Jun 2020 01:01:18 +0000
> +
>  xfsdump (3.1.9) unstable; urgency=low
>  
>    * New upstream release
> diff -Nru xfsdump-3.1.9/debian/rules xfsdump-3.1.9+nmu1/debian/rules
> --- xfsdump-3.1.9/debian/rules	2020-01-31 17:30:58.000000000 +0000
> +++ xfsdump-3.1.9+nmu1/debian/rules	2020-06-19 01:01:18.000000000 +0000
> @@ -44,6 +44,9 @@
>  	-rm -rf $(dirme)
>  	$(pkgme) $(MAKE) -C . install
>  	$(pkgme) $(MAKE) dist
> +	#remove the symlinks in /usr/sbin, the postinst will create them
> +	#if appropriate for the users system 
> +	rm -f debian/xfsdump/usr/sbin/xfsdump debian/xfsdump/usr/sbin/xfsrestore
>  	dh_installdocs
>  	dh_installchangelogs
>  	dh_strip
> diff -Nru xfsdump-3.1.9/debian/xfsdump.postinst xfsdump-3.1.9+nmu1/debian/xfsdump.postinst
> --- xfsdump-3.1.9/debian/xfsdump.postinst	1970-01-01 00:00:00.000000000 +0000
> +++ xfsdump-3.1.9+nmu1/debian/xfsdump.postinst	2020-06-19 00:59:32.000000000 +0000
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +set -e
> +
> +if [ "$1" = 'configure' ]; then
> +  for file in xfsdump xfsrestore; do
> +    if [ ! -e /usr/sbin/$file ]; then
> +      ln -s /sbin/$file /usr/sbin/$file
> +    fi
> +  done
> +fi
> +
> +#DEBHELPER#
> diff -Nru xfsdump-3.1.9/debian/xfsdump.preinst xfsdump-3.1.9+nmu1/debian/xfsdump.preinst
> --- xfsdump-3.1.9/debian/xfsdump.preinst	1970-01-01 00:00:00.000000000 +0000
> +++ xfsdump-3.1.9+nmu1/debian/xfsdump.preinst	2020-06-19 01:01:18.000000000 +0000
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +set -e
> +
> +if [ "$1" = 'remove' ]; then
> +  for file in xfsdump xfsrestore; do
> +    if [ -L /usr/sbin/$file ]; then
> +      rm /usr/sbin/$file
> +    fi
> +  done
> +fi
> +
> +#DEBHELPER#

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[Eric Sandeen]

On 6/18/20 11:47 PM, Darrick J. Wong wrote:
> On Fri, Jun 19, 2020 at 05:05:00AM +0100, peter green wrote:
>> (original message was sent to nathans@redhat.com
>> 953537@bugs.debian.org and linux-xfs@vger.kernel.org re-sending as
>> plain-text only to linux-xfs@vger.kernel.org)
>>
>> This bug has now caused xfsdump to be kicked out of testing which is
>> making amanda unbuildable in testing.
> 
> Uhoh...
> 
>>
>>
>>> Yes, what's really needed here is for a change to be merged upstream
>>> (as all other deb packaging artifacts are) otherwise this will keep
>>> getting lost in time.
>> To make it easier to upstream this I whipped up a patch that should
>> solve the issue while only modifying the debian packaging and not
>> touching the upstream makefiles. It is attached to this message and if
>> I get no response I will likely do some further testing and then NMU
>> it in Debian.
>>
>> One issue I noticed is it's not all all obvious who upstream is. The
>> sgi website listed in README seems to be long dead and there are no
>> obvious upstream results in a google search for xfsdump. Gentoos page
>> on xfsdump links to https://xfs.wiki.kernel.org but that page makes no
>> mention of xfsdump.
>>
>> I eventually poked around on git.kernel.org and my best guess is that
>> https://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git/ is the upstream
>> git repository and linux-xfs@vger.kernel.org is the appropriate
>> mailing list, I would appreciate comments on whether or not this is
>> correct and updates to the documentation to reflect whatever the
>> correct location is.
> 
> Yep, you've found us. :)
> 
> Uh... seeing how /sbin seems to be a symlink to /usr/sbin on more and
> more distros now, how about we just change the upstream makefile to dump
> them in /usr/sbin and forget all about the symlinks?
> 
> (He says, wondering what the actual maintainer will say...)

I wonder too :P

So, FWIW, fedora/rhel packaging also hacks this up :(

xfsdump does:

%install
rm -rf $RPM_BUILD_ROOT
make DIST_ROOT=$RPM_BUILD_ROOT install
# remove non-versioned docs location
rm -rf $RPM_BUILD_ROOT/%{_datadir}/doc/xfsdump/

# Bit of a hack to move files from /sbin to /usr/sbin
(cd $RPM_BUILD_ROOT/%{_sbindir}; rm xfsdump xfsrestore)
(cd $RPM_BUILD_ROOT/%{_sbindir}; mv ../../sbin/xfsdump .)
(cd $RPM_BUILD_ROOT/%{_sbindir}; mv ../../sbin/xfsrestore .)

xfsprogs does:

%install
make DIST_ROOT=$RPM_BUILD_ROOT install install-dev \
        PKG_ROOT_SBIN_DIR=%{_sbindir} PKG_ROOT_LIB_DIR=%{_libdir}

Both of these work around the default location of /sbin:

# grep PKG_ROOT_SBIN_DIR xfsprogs-maint/include/builddefs xfsdump/include/builddefs
xfsprogs-maint/include/builddefs:PKG_ROOT_SBIN_DIR = /sbin
xfsdump/include/builddefs:PKG_ROOT_SBIN_DIR = /sbin

On one hand, it'd be easy enough to change the upstream defaults I guess.
On the other hand, I think the PKG_ROOT_SBIN_DIR method is easy too.

How does debian fix this for xfsprogs?  Doesn't the same issue exist?

-Eric

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[Dave Chinner]

On Fri, Jun 19, 2020 at 12:03:35PM -0500, Eric Sandeen wrote:
> On 6/18/20 11:47 PM, Darrick J. Wong wrote:
> > On Fri, Jun 19, 2020 at 05:05:00AM +0100, peter green wrote:
> >> (original message was sent to nathans@redhat.com
> >> 953537@bugs.debian.org and linux-xfs@vger.kernel.org re-sending as
> >> plain-text only to linux-xfs@vger.kernel.org)
> >>
> >> This bug has now caused xfsdump to be kicked out of testing which is
> >> making amanda unbuildable in testing.
> > 
> > Uhoh...
> > 
> >>
> >>
> >>> Yes, what's really needed here is for a change to be merged upstream
> >>> (as all other deb packaging artifacts are) otherwise this will keep
> >>> getting lost in time.
> >> To make it easier to upstream this I whipped up a patch that should
> >> solve the issue while only modifying the debian packaging and not
> >> touching the upstream makefiles. It is attached to this message and if
> >> I get no response I will likely do some further testing and then NMU
> >> it in Debian.
> >>
> >> One issue I noticed is it's not all all obvious who upstream is. The
> >> sgi website listed in README seems to be long dead and there are no
> >> obvious upstream results in a google search for xfsdump. Gentoos page
> >> on xfsdump links to https://xfs.wiki.kernel.org but that page makes no
> >> mention of xfsdump.
> >>
> >> I eventually poked around on git.kernel.org and my best guess is that
> >> https://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git/ is the upstream
> >> git repository and linux-xfs@vger.kernel.org is the appropriate
> >> mailing list, I would appreciate comments on whether or not this is
> >> correct and updates to the documentation to reflect whatever the
> >> correct location is.
> > 
> > Yep, you've found us. :)
> > 
> > Uh... seeing how /sbin seems to be a symlink to /usr/sbin on more and
> > more distros now, how about we just change the upstream makefile to dump
> > them in /usr/sbin and forget all about the symlinks?
> > 
> > (He says, wondering what the actual maintainer will say...)
> 
> I wonder too :P
> 
> So, FWIW, fedora/rhel packaging also hacks this up :(

Isn't the configure script supposed to handle install locations?
Both xfsprogs and xfsdump have this in their include/builddefs.in:

PKG_ROOT_SBIN_DIR = @root_sbindir@
PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@

So the actual install locations are coming from the autoconf setup
the build runs under. Looking in configure.ac in xfsprogs and
xfsdump, they both do the same thing:

.....
#
# Some important tools should be installed into the root partitions.
#
# Check whether exec_prefix=/usr: and install them to /sbin in that
# case.  If the user choses a different prefix assume he just wants
# a local install for testing and not a system install.
#
case $exec_prefix:$prefix in
NONE:NONE | NONE:/usr | /usr:*)
  root_sbindir='/sbin'
  root_libdir="/${base_libdir}"
  ;;
*)
  root_sbindir="${sbindir}"
  root_libdir="${libdir}"
  ;;
esac

AC_SUBST([root_sbindir])
AC_SUBST([root_libdir])

....

I suspect that this "system install" logic - which once made sense -
doesn't work at all with symlinked /sbin setups.

IIRC debian is in a transistion stage where it will accept either
types of package, but people trying to install a "linked /sbin only"
system will be reporting issues where pacakges do the wrong thing...

> xfsdump does:
> 
> %install
> rm -rf $RPM_BUILD_ROOT
> make DIST_ROOT=$RPM_BUILD_ROOT install
> # remove non-versioned docs location
> rm -rf $RPM_BUILD_ROOT/%{_datadir}/doc/xfsdump/
> 
> # Bit of a hack to move files from /sbin to /usr/sbin
> (cd $RPM_BUILD_ROOT/%{_sbindir}; rm xfsdump xfsrestore)
> (cd $RPM_BUILD_ROOT/%{_sbindir}; mv ../../sbin/xfsdump .)
> (cd $RPM_BUILD_ROOT/%{_sbindir}; mv ../../sbin/xfsrestore .)
> 
> xfsprogs does:
> 
> %install
> make DIST_ROOT=$RPM_BUILD_ROOT install install-dev \
>         PKG_ROOT_SBIN_DIR=%{_sbindir} PKG_ROOT_LIB_DIR=%{_libdir}

So the fedora rpm package build is overriding the locations that
autoconf set in include/builddefs for the install on the make
command line?

> Both of these work around the default location of /sbin:
> 
> # grep PKG_ROOT_SBIN_DIR xfsprogs-maint/include/builddefs xfsdump/include/builddefs
> xfsprogs-maint/include/builddefs:PKG_ROOT_SBIN_DIR = /sbin
> xfsdump/include/builddefs:PKG_ROOT_SBIN_DIR = /sbin

Ok, it is.

So my question is this: what magic should we be putting in autoconf
to have it automatically detect that the package should build for a
linked /sbin and have the build always do the right thing via "make
install"?

> On one hand, it'd be easy enough to change the upstream defaults I guess.
> On the other hand, I think the PKG_ROOT_SBIN_DIR method is easy too.
> 
> How does debian fix this for xfsprogs?  Doesn't the same issue exist?

AFAICT, yes, it does exist. The buildroot from a recent package
build:

$ ls -l xfsprogs-5.7.0-rc0/debian/xfsprogs/sbin
total 928
-rwxr-xr-x 1 dave dave   1968 May 21 15:41 fsck.xfs
-rwxr-xr-x 1 dave dave 367584 May 21 15:42 mkfs.xfs
-rwxr-xr-x 1 dave dave 573536 May 21 15:42 xfs_repair
$

xfsprogs also appears to be packaging binaries in /sbin, too.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[peter green]

Putting the debian bug back in cc, previous mails are visible at https://marc.info/?l=linux-xfs&m=159253950420613&w=2

On 19/06/2020 23:43, Dave Chinner wrote:

> Isn't the configure script supposed to handle install locations?
> Both xfsprogs and xfsdump have this in their include/builddefs.in:
> 
> PKG_ROOT_SBIN_DIR = @root_sbindir@
> PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@
> 
> So the actual install locations are coming from the autoconf setup
> the build runs under. Looking in configure.ac in xfsprogs and
> xfsdump, they both do the same thing:
> 
The issue is that xfsdump installs the programs in /sbin but it *also* creates symlinks in /usr/sbin,
presumablly at some point the binaries were moved to /sbin but the developers wanted to keep
packages that hardcoded the paths working.

Those symlinks are suppressed if installing directly to a merged-usr system, which is fine for
end-users installing the program directly but isn't useful if installing to a destination dir for
packaing purposed.

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[peter green]

Putting the Debian bug back in cc, for earlier mails please see https://marc.info/?l=linux-xfs&m=159253950420613&w=2
Eric Sandeen wrote:
> 
> How does debian fix this for xfsprogs?  Doesn't the same issue exist?


I'm not seeing any cases like xfsdump where a binary is located in /sbin but symlinked from /usr/sbin .

Debian merged-usr systems can deal with files in /sbin and files in /usr/sbin, what needs special
treatment is filenames that exist in both.

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[peter green]

On 19/06/2020 05:47, Darrick J. Wong wrote:

>> I eventually poked around on git.kernel.org and my best guess is that
>> https://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git/ is the upstream
>> git repository and linux-xfs@vger.kernel.org is the appropriate
>> mailing list, I would appreciate comments on whether or not this is
>> correct and updates to the documentation to reflect whatever the
>> correct location is.
> 
> Yep, you've found us. :)
> 

Would it be too much to ask that the wiki is updated to mention xfsdump
and the README is updated so that people can find you more easilly in
future?

Re: Bug#953537: xfsdump fails to install in /usr merged system.

[Dave Chinner]

On Sat, Jun 20, 2020 at 03:10:05AM +0100, peter green wrote:
> Putting the debian bug back in cc, previous mails are visible at https://marc.info/?l=linux-xfs&m=159253950420613&w=2
> 
> On 19/06/2020 23:43, Dave Chinner wrote:
> 
> > Isn't the configure script supposed to handle install locations?
> > Both xfsprogs and xfsdump have this in their include/builddefs.in:
> > 
> > PKG_ROOT_SBIN_DIR = @root_sbindir@
> > PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@
> > 
> > So the actual install locations are coming from the autoconf setup
> > the build runs under. Looking in configure.ac in xfsprogs and
> > xfsdump, they both do the same thing:
> > 
> The issue is that xfsdump installs the programs in /sbin but it *also* creates symlinks in /usr/sbin,
> presumablly at some point the binaries were moved to /sbin but the developers wanted to keep
> packages that hardcoded the paths working.

Ah, thanks for explaining the issue, Peter.

> Those symlinks are suppressed if installing directly to a merged-usr system, which is fine for
> end-users installing the program directly but isn't useful if installing to a destination dir for
> packaing purposed.

OK, so what I'm really missing here is knowledge on how a modern
distro is expecting binaries to be packaged. If every distro is
having to work around the install locations the package uses by
default, then we need to fix up the default locations and behaviour
of the upstream packaging.

So, what is expected behaviour in these situations:

	- user installs directly in /usr/local/... or $HOME/...
	- user installs directly in /usr/....
	- package installed on split-usr system
	- package installed on merged-usr system

Th first two here are coded into the build via 'make install', and
the package build systems use the same install command to build the
package. It seems to me that we'd want 'make install' to behave the
same way for the last 3 cases, but I'm missing how the
build/packaging system is actually supposed to reliably detect the
differences between such split/merged systems and hence just DTRT....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com