[PATCH 0/4] xfsprogs: packaging cleanups

[PATCH 0/4] xfsprogs: packaging cleanups

[Darrick J. Wong]

Hi all,

Introduce flatpak build system support, forward-port debian packages to
compat 11 to take advantage of systemd helper scripts.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=packaging-cleanups

[PATCH 1/4] misc: support building flatpak images

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Poke the system until it supports building flatpaks.  Maybe.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 bootstrap  |   12 ++++++++++++
 flatpak.sh |   55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100755 bootstrap
 create mode 100755 flatpak.sh


diff --git a/bootstrap b/bootstrap
new file mode 100755
index 00000000..d403cf58
--- /dev/null
+++ b/bootstrap
@@ -0,0 +1,12 @@
+#!/bin/bash -x
+
+libtoolize
+automake --add-missing
+aclocal -I m4
+autoconf
+make configure
+libtoolize
+automake --add-missing
+aclocal -I m4
+autoconf
+make configure
diff --git a/flatpak.sh b/flatpak.sh
new file mode 100755
index 00000000..a1ec916e
--- /dev/null
+++ b/flatpak.sh
@@ -0,0 +1,55 @@
+#!/bin/bash -xe
+
+if [ "$1" = "--help" ]; then
+	echo "Usage: $0 [branch] [build]"
+	echo "Build xfsprogs flatpak from the given branch."
+fi
+
+branch="$1"
+test -z "${branch}" && branch=master
+build="$2"
+test -z "${build}" && build=release
+
+if [ "${build}" = "debug" ]; then
+	config_opts=', "--disable-lto"'
+fi
+
+cat > flatpak.ini << ENDL
+[Application]
+name=org.xfs.progs
+runtime=org.freedesktop.Platform/$(uname -m)/18.08
+
+[Context]
+devices=all;
+ENDL
+
+cat > org.xfs.progs.json << ENDL
+{
+    "app-id": "org.xfs.progs",
+    "runtime": "org.freedesktop.Platform",
+    "runtime-version": "18.08",
+    "sdk": "org.freedesktop.Sdk",
+    "command": "/app/sbin/xfs_repair",
+    "metadata": "flatpak.ini",
+    "modules": [
+        {
+            "name": "xfsprogs",
+            "buildsystem": "autotools",
+            "config-opts": ["--enable-shared=no" ${config_opts} ],
+            "sources": [
+                {
+                    "type": "git",
+                    "path": "${PWD}/",
+                    "branch": "${branch}"
+                }
+            ]
+        }
+    ]
+}
+ENDL
+
+flatpak-builder --repo=repo build-dir org.xfs.progs.json --force-clean
+flatpak build-bundle repo xfsprogs.flatpak org.xfs.progs
+echo build done, run:
+echo flatpak install xfsprogs.flatpak
+echo flatpak run org.xfs.progs

Re: [PATCH 1/4] misc: support building flatpak images

[Christoph Hellwig]

On Tue, Dec 31, 2019 at 05:22:47PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Poke the system until it supports building flatpaks.  Maybe.

Why would we want to support such a fucked up package delivery
mechanism?

Re: [PATCH 1/4] misc: support building flatpak images

[Darrick J. Wong]

On Tue, Jan 07, 2020 at 06:20:35AM -0800, Christoph Hellwig wrote:
> On Tue, Dec 31, 2019 at 05:22:47PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Poke the system until it supports building flatpaks.  Maybe.
> 
> Why would we want to support such a fucked up package delivery
> mechanism?

In theory we could use it as a means to distribute uptodate xfs_repair
so that anyone with a problem that their distro's copy of xfs_repair
can't or won't fix can try upstream without having to build xfsprogs
themselves.

Though I wonder how many people in that situation are running some
colorful linux distribution but are too stingy to pay for a support
contract.... :)

Also, can you elaborate on 'fucked up package delivery mechanism'?

Admittedly I'm /not/ volunteering to maintain a xfs_repair flatpak, so
perhaps I should just drop this whole thing.

--D

Re: [PATCH 1/4] misc: support building flatpak images

[Christoph Hellwig]

On Tue, Jan 07, 2020 at 11:35:37AM -0800, Darrick J. Wong wrote:
> In theory we could use it as a means to distribute uptodate xfs_repair
> so that anyone with a problem that their distro's copy of xfs_repair
> can't or won't fix can try upstream without having to build xfsprogs
> themselves.

And someone would allow a random weirdo package format access to their
disks?

> Also, can you elaborate on 'fucked up package delivery mechanism'?

It completely messes up how the system is supposed to work, and not your
are in a maze of unssuportable package versions just like npm.

[PATCH 2/4] debian: turn debhelper compat level up to 11

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Upgrade to debhelper level 11 to take advantage of dh_installsystemd,
which greatly simplifies the installation and activation of the scrub
systemd services.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/compat |    2 +-
 debian/rules  |    1 +
 2 files changed, 2 insertions(+), 1 deletion(-)


diff --git a/debian/compat b/debian/compat
index ec635144..b4de3947 100644
--- a/debian/compat
+++ b/debian/compat
@@ -1 +1 @@
-9
+11
diff --git a/debian/rules b/debian/rules
index e8509fb3..a79db896 100755
--- a/debian/rules
+++ b/debian/rules
@@ -93,6 +93,7 @@ binary-arch: checkroot built
 	dh_compress
 	dh_fixperms
 	dh_makeshlibs
+	dh_installsystemd -p xfsprogs --no-enable --no-start --no-restart-after-upgrade --no-stop-on-upgrade
 	dh_installdeb
 	dh_shlibdeps
 	dh_gencontrol

Re: [PATCH 2/4] debian: turn debhelper compat level up to 11

[Christoph Hellwig]

On Tue, Dec 31, 2019 at 05:22:53PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Upgrade to debhelper level 11 to take advantage of dh_installsystemd,
> which greatly simplifies the installation and activation of the scrub
> systemd services.

The subject and description seem upside down.  You want to use
dh_installsystemd, and to do so you'll have to bump the level..

Re: [PATCH 2/4] debian: turn debhelper compat level up to 11

[Darrick J. Wong]

On Tue, Jan 07, 2020 at 06:21:23AM -0800, Christoph Hellwig wrote:
> On Tue, Dec 31, 2019 at 05:22:53PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Upgrade to debhelper level 11 to take advantage of dh_installsystemd,
> > which greatly simplifies the installation and activation of the scrub
> > systemd services.
> 
> The subject and description seem upside down.  You want to use
> dh_installsystemd, and to do so you'll have to bump the level..

Heh, ok.

"debian: install scrub services with dh_installsystemd

"Use dh_installsystemd to handle the installation and activation of the
scrub systemd services.  This requires bumping the compat version to
11."

(Admittedly the subject was a riff on the old 'turn it up to 11' joke)

Related question: Are Debian oldoldstable and Ubuntu 16.04LTS old enough
to drop the "debian: permit compat level 9 dh builds" patch?  On systems
with software that old the kernel isn't going to support scrub, and who's
going to build xfsprogs 5.xx for those old systems?

--D

Re: [PATCH 2/4] debian: turn debhelper compat level up to 11

[Christoph Hellwig]

On Tue, Jan 07, 2020 at 10:51:51AM -0800, Darrick J. Wong wrote:
> Related question: Are Debian oldoldstable and Ubuntu 16.04LTS old enough
> to drop the "debian: permit compat level 9 dh builds" patch?  On systems
> with software that old the kernel isn't going to support scrub, and who's
> going to build xfsprogs 5.xx for those old systems?

This sounds old enough to not bother.  And if someone screams loud
enough we could still add it back.

[PATCH 3/4] debian: permit compat level 9 dh builds

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Retain compatibility with existing dh level 9 build systems.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/rules |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)


diff --git a/debian/rules b/debian/rules
index a79db896..1dd1f524 100755
--- a/debian/rules
+++ b/debian/rules
@@ -1,5 +1,7 @@
 #!/usr/bin/make -f
 
+.PHONY: dpkg_config
+
 export DH_VERBOSE=1
 
 ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
@@ -7,6 +9,9 @@ ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
     PMAKEFLAGS += -j$(NUMJOBS)
 endif
 
+DH_VERSION := $(shell dpkg-query -W -f '$${Version}' debhelper)
+USE_DH9 ?= $(shell if dpkg --compare-versions $(DH_VERSION) lt 11 ; then echo yes ; fi)
+
 package = xfsprogs
 develop = xfslibs-dev
 bootpkg = xfsprogs-udeb
@@ -33,11 +38,18 @@ checkdir = test -f debian/rules
 build: build-arch build-indep
 build-arch: built
 build-indep: built
-built: dibuild config
+built: dpkg_config dibuild config
 	@echo "== dpkg-buildpackage: build" 1>&2
 	$(MAKE) $(PMAKEFLAGS) default
 	touch built
 
+dpkg_config:
+ifeq ($(USE_DH9),yes)
+	mv debian/compat debian/compat.save
+	echo 9 > debian/compat
+endif
+	@true
+
 config: .census
 .census:
 	@echo "== dpkg-buildpackage: configure" 1>&2
@@ -93,7 +105,9 @@ binary-arch: checkroot built
 	dh_compress
 	dh_fixperms
 	dh_makeshlibs
+ifneq ($(USE_DH9),yes)
 	dh_installsystemd -p xfsprogs --no-enable --no-start --no-restart-after-upgrade --no-stop-on-upgrade
+endif
 	dh_installdeb
 	dh_shlibdeps
 	dh_gencontrol

[PATCH 4/4] xfs_scrub_all: failure reporting for the xfs_scrub_all job

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Create a failure reporting service for when xfs_scrub_all fails.  This
is probably a debug-only patch.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/Makefile                      |    3 ++-
 scrub/xfs_scrub_all_fail            |   24 ++++++++++++++++++++++++
 scrub/xfs_scrub_all_fail.service.in |   11 +++++++++++
 3 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100755 scrub/xfs_scrub_all_fail
 create mode 100644 scrub/xfs_scrub_all_fail.service.in


diff --git a/scrub/Makefile b/scrub/Makefile
index 47c887eb..50d16121 100644
--- a/scrub/Makefile
+++ b/scrub/Makefile
@@ -18,7 +18,7 @@ XFS_SCRUB_ALL_PROG = xfs_scrub_all
 XFS_SCRUB_ARGS = -b -n
 ifeq ($(HAVE_SYSTEMD),yes)
 INSTALL_SCRUB += install-systemd
-SYSTEMD_SERVICES = xfs_scrub@.service xfs_scrub_all.service xfs_scrub_all.timer xfs_scrub_fail@.service
+SYSTEMD_SERVICES = xfs_scrub@.service xfs_scrub_all.service xfs_scrub_all.timer xfs_scrub_fail@.service xfs_scrub_all_fail.service
 OPTIONAL_TARGETS += $(SYSTEMD_SERVICES)
 endif
 ifeq ($(HAVE_CROND),yes)
@@ -133,6 +133,7 @@ install-systemd: default $(SYSTEMD_SERVICES)
 	$(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR)
 	$(INSTALL) -m 755 -d $(PKG_LIB_DIR)/$(PKG_NAME)
 	$(INSTALL) -m 755 xfs_scrub_fail $(PKG_LIB_DIR)/$(PKG_NAME)
+	$(INSTALL) -m 755 xfs_scrub_all_fail $(PKG_LIB_DIR)/$(PKG_NAME)
 
 install-crond: default $(CRONTABS)
 	$(INSTALL) -m 755 -d $(CROND_DIR)
diff --git a/scrub/xfs_scrub_all_fail b/scrub/xfs_scrub_all_fail
new file mode 100755
index 00000000..8391abc8
--- /dev/null
+++ b/scrub/xfs_scrub_all_fail
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Email logs of failed xfs_scrub_all unit runs
+
+mailer=/usr/sbin/sendmail
+recipient="$1"
+test -z "${recipient}" && exit 0
+hostname="$(hostname -f 2>/dev/null)"
+test -z "${hostname}" && hostname="${HOSTNAME}"
+if [ ! -x "${mailer}" ]; then
+	echo "${mailer}: Mailer program not found."
+	exit 1
+fi
+
+(cat << ENDL
+To: $1
+From: <xfs_scrub_all@${hostname}>
+Subject: xfs_scrub_all failure on ${mntpoint}
+
+So sorry, the automatic xfs_scrub_all on ${hostname} failed.
+
+A log of what happened follows:
+ENDL
+systemctl status --full --lines 4294967295 "xfs_scrub_all") | "${mailer}" -t -i
diff --git a/scrub/xfs_scrub_all_fail.service.in b/scrub/xfs_scrub_all_fail.service.in
new file mode 100644
index 00000000..2d85db28
--- /dev/null
+++ b/scrub/xfs_scrub_all_fail.service.in
@@ -0,0 +1,11 @@
+[Unit]
+Description=Online XFS Metadata Check for All Filesystems Failure Reporting for %I
+Documentation=man:xfs_scrub_all(8)
+
+[Service]
+Type=oneshot
+Environment=EMAIL_ADDR=root
+ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_all_fail "${EMAIL_ADDR}"
+User=mail
+Group=mail
+SupplementaryGroups=systemd-journal

[PATCHSET v24.0 0/4] xfs_scrub_all: automatic media scan service

[Darrick J. Wong]

Hi all,

Now that we've completed the online fsck functionality, there are a few
things that could be improved in the automatic service.  Specifically,
we would like to perform a more intensive metadata + media scan once per
month, to give the user confidence that the filesystem isn't losing data
silently.  To accomplish this, enhance xfs_scrub_all to be able to
trigger media scans.  Next, add a duplicate set of system services that
start the media scans automatically.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service
---
 debian/rules                           |    3 +
 include/builddefs.in                   |    3 +
 man/man8/Makefile                      |    7 ++
 man/man8/xfs_scrub_all.8.in            |   20 ++++++
 scrub/Makefile                         |   14 ++++-
 scrub/xfs_scrub_all.cron.in            |    2 -
 scrub/xfs_scrub_all.in                 |   99 +++++++++++++++++++++++++++++++-
 scrub/xfs_scrub_all.service.in         |    9 ++-
 scrub/xfs_scrub_all_fail.service.in    |   67 ++++++++++++++++++++++
 scrub/xfs_scrub_fail                   |   44 +++++++++++---
 scrub/xfs_scrub_fail@.service.in       |    2 -
 scrub/xfs_scrub_media@.service.in      |   90 +++++++++++++++++++++++++++++
 scrub/xfs_scrub_media_fail@.service.in |   76 +++++++++++++++++++++++++
 13 files changed, 413 insertions(+), 23 deletions(-)
 rename man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} (59%)
 create mode 100644 scrub/xfs_scrub_all_fail.service.in
 create mode 100644 scrub/xfs_scrub_media@.service.in
 create mode 100644 scrub/xfs_scrub_media_fail@.service.in

[PATCH 4/4] xfs_scrub_all: failure reporting for the xfs_scrub_all job

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Create a failure reporting service for when xfs_scrub_all fails.  This
shouldn't happen often, but let's report anyways.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 scrub/Makefile                      |    1 +
 scrub/xfs_scrub_all.service.in      |    1 +
 scrub/xfs_scrub_all_fail.service.in |   67 +++++++++++++++++++++++++++++++++++
 scrub/xfs_scrub_fail                |   33 ++++++++++++++---
 4 files changed, 96 insertions(+), 6 deletions(-)
 create mode 100644 scrub/xfs_scrub_all_fail.service.in


diff --git a/scrub/Makefile b/scrub/Makefile
index 7434ac0ce4e..f2d0c1aa0bf 100644
--- a/scrub/Makefile
+++ b/scrub/Makefile
@@ -21,6 +21,7 @@ SYSTEMD_SERVICES=\
 	xfs_scrub_media@.service \
 	xfs_scrub_media_fail@.service \
 	xfs_scrub_all.service \
+	xfs_scrub_all_fail.service \
 	xfs_scrub_all.timer \
 	system-xfs_scrub.slice
 OPTIONAL_TARGETS += $(SYSTEMD_SERVICES)
diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in
index e831ad58eb8..c11d6a96037 100644
--- a/scrub/xfs_scrub_all.service.in
+++ b/scrub/xfs_scrub_all.service.in
@@ -5,6 +5,7 @@
 
 [Unit]
 Description=Online XFS Metadata Check for All Filesystems
+OnFailure=xfs_scrub_all_fail.service
 ConditionACPower=true
 Documentation=man:xfs_scrub_all(8)
 After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
diff --git a/scrub/xfs_scrub_all_fail.service.in b/scrub/xfs_scrub_all_fail.service.in
new file mode 100644
index 00000000000..002d05b67de
--- /dev/null
+++ b/scrub/xfs_scrub_all_fail.service.in
@@ -0,0 +1,67 @@
+[Unit]
+Description=Online XFS Metadata Check for All Filesystems Failure Reporting
+Documentation=man:xfs_scrub_all(8)
+
+[Service]
+Type=oneshot
+Environment=EMAIL_ADDR=root
+ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" xfs_scrub_all
+User=mail
+Group=mail
+SupplementaryGroups=systemd-journal
+
+# No realtime scheduling
+RestrictRealtime=true
+
+# Make the entire filesystem readonly and /home inaccessible, then bind mount
+# the filesystem we're supposed to be checking into our private /tmp dir.
+ProtectSystem=full
+ProtectHome=yes
+PrivateTmp=true
+RestrictSUIDSGID=true
+
+# Emailing reports requires network access, but not the ability to change the
+# hostname.
+ProtectHostname=true
+
+# Don't let the program mess with the kernel configuration at all
+ProtectKernelLogs=true
+ProtectKernelModules=true
+ProtectKernelTunables=true
+ProtectControlGroups=true
+ProtectProc=invisible
+RestrictNamespaces=true
+
+# Can't hide /proc because journalctl needs it to find various pieces of log
+# information
+#ProcSubset=pid
+
+# Only allow the default personality Linux
+LockPersonality=true
+
+# No writable memory pages
+MemoryDenyWriteExecute=true
+
+# Don't let our mounts leak out to the host
+PrivateMounts=true
+
+# Restrict system calls to the native arch and only enough to get things going
+SystemCallArchitectures=native
+SystemCallFilter=@system-service
+SystemCallFilter=~@privileged
+SystemCallFilter=~@resources
+SystemCallFilter=~@mount
+
+# xfs_scrub needs these privileges to run, and no others
+CapabilityBoundingSet=
+NoNewPrivileges=true
+
+# Failure reporting shouldn't create world-readable files
+UMask=0077
+
+# Clean up any IPC objects when this unit stops
+RemoveIPC=true
+
+# No access to hardware device files
+PrivateDevices=true
+ProtectClock=true
diff --git a/scrub/xfs_scrub_fail b/scrub/xfs_scrub_fail
index 58c50abe963..0739a00dec9 100755
--- a/scrub/xfs_scrub_fail
+++ b/scrub/xfs_scrub_fail
@@ -5,14 +5,13 @@
 # Copyright (C) 2018 Oracle.  All Rights Reserved.
 # Author: Darrick J. Wong <djwong@kernel.org>
 
-# Email logs of failed xfs_scrub unit runs
+# Email logs of failed xfs_scrub and xfs_scrub_all unit runs
 
 recipient="$1"
 test -z "${recipient}" && exit 0
 service="$2"
 test -z "${service}" && exit 0
 mntpoint="$3"
-test -z "${mntpoint}" && exit 0
 
 hostname="$(hostname -f 2>/dev/null)"
 test -z "${hostname}" && hostname="${HOSTNAME}"
@@ -47,10 +46,12 @@ escape_path() {
 	echo "-$(systemd-escape --path "${mntpoint}")"
 }
 
-mntpoint_esc="$(escape_path "${mntpoint}")"
+fail_mail_mntpoint() {
+	local mntpoint_esc
 
-(cat << ENDL
-To: $1
+	mntpoint_esc="$(escape_path "${mntpoint}")"
+	cat << ENDL
+To: ${recipient}
 From: <${service}@${hostname}>
 Subject: ${service} failure on ${mntpoint}
 
@@ -58,5 +59,25 @@ So sorry, the automatic ${service} of ${mntpoint} on ${hostname} failed.
 
 A log of what happened follows:
 ENDL
-systemctl status --full --lines 4294967295 "${service}@${mntpoint_esc}") | "${mailer}" -t -i
+	systemctl status --full --lines 4294967295 "${service}@${mntpoint_esc}"
+}
+
+fail_mail() {
+	cat << ENDL
+To: ${recipient}
+From: <${service}@${hostname}>
+Subject: ${service} failure
+
+So sorry, the automatic ${service} on ${hostname} failed.
+
+A log of what happened follows:
+ENDL
+	systemctl status --full --lines 4294967295 "${service}"
+}
+
+if [ -n "${mntpoint}" ]; then
+	fail_mail_mntpoint | "${mailer}" -t -i
+else
+	fail_mail | "${mailer}" -t -i
+fi
 exit "${PIPESTATUS[1]}"