* [PATCH 00/18] mirrors: enhance and add tons of mirrors
@ 2024-03-30 2:25 Luis Chamberlain
2024-03-30 2:25 ` [PATCH 01/18] linux-mirror: add kdevops mirroring support Luis Chamberlain
` (18 more replies)
0 siblings, 19 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
If you are stuck in a world where you don't have much data for your
downloads you have only one option: mirror everything. So the motivation
for this is the exand the existing mirror support on kdevops to mirror
more things other than just the kernel and qemu. We now can mirror
*everything* we clone including kdevops itself.
We also enhance this all so that we can *use* the mirrors by *default*
if you have them and you are using local virtualization. We differentiate
between vagrant and guestfs.
Towards the end this just simplifies the process by adding a yaml file
for the mirrors so that we can ingest them and process that with a few
python scripts. The last commits demonstrate how easy it is now to add
new mirrors.
There's only few mirrors left to add but we can do that next week.
Luis Chamberlain (18):
linux-mirror: add kdevops mirroring support
Makefile.min_deps: add netcat as a min binary dependency
provision: generalize bridge-ip further
mirrors: add guestfs mirror options without 9p
mirrors: enable kdevops mirror
mirrors: add fstests to mirrors
fstests: use local primary fstests mirror if present
mirrors: add kdevops git alternative for fstests
mirrors: use kdevops fstests mirror when available
mirrors: add blktests mirror
mirrors: use blktests mirror when its available
mirrors: increase scope of mirroring in output
mirrors: move status check
mirrors: move mirror editing into one file
mirrors: add new mirrors.yaml and add xfsprogs
mirrors: add xfsdump mirror and use it
mirrors: add dbench git
mirrors: add blktrace mirror and use it
.gitignore | 5 +
Kconfig | 1 +
Makefile | 2 -
Makefile.linux-mirror | 55 ++-----
Makefile.min_deps | 1 +
kconfigs/Kconfig.ansible_provisioning | 4 +-
kconfigs/Kconfig.defaults | 18 +++
kconfigs/Kconfig.guestfs | 6 -
kconfigs/Kconfig.mirror | 115 +++++++++++++-
kconfigs/workflows/Kconfig.shared | 11 +-
playbooks/linux-mirror.yml | 4 +
playbooks/roles/blktests/defaults/main.yml | 1 +
playbooks/roles/blktests/tasks/main.yml | 2 +-
playbooks/roles/fstests/defaults/main.yml | 2 +
.../tasks/install-deps/debian/main.yml | 2 +-
playbooks/roles/linux-mirror/README.md | 9 ++
.../roles/linux-mirror/defaults/main.yml | 8 +
.../linux-mirror-systemd/Makefile | 62 +++++++-
.../blktests-mirror.service | 13 ++
.../blktests-mirror.timer | 11 ++
.../fstests-mirror.service | 13 ++
.../linux-mirror-systemd/fstests-mirror.timer | 11 ++
.../kdevops-fstests-mirror.service | 13 ++
.../kdevops-fstests-mirror.timer | 11 ++
.../kdevops-mirror.service | 13 ++
.../linux-mirror-systemd/kdevops-mirror.timer | 10 ++
.../linux-mirror/python/gen-mirror-files.py | 149 ++++++++++++++++++
.../linux-mirror/python/start-mirroring.py | 110 +++++++++++++
playbooks/roles/linux-mirror/tasks/main.yml | 91 +++++++++++
.../linux-mirror/templates/mirrors.yaml.j2 | 13 ++
scripts/append-makefile-vars.sh | 10 ++
scripts/check_mirror_present.sh | 11 ++
scripts/get-distro-default-bridge.sh | 15 ++
scripts/get-distro-prefix.sh | 18 +++
scripts/provision.Makefile | 12 ++
workflows/blktests/Kconfig | 29 +++-
workflows/blktests/Makefile | 1 +
workflows/fstests/Kconfig | 20 ++-
workflows/fstests/Makefile.xfs | 2 +
workflows/fstests/xfs/Kconfig | 20 ++-
workflows/linux/Kconfig.linus | 3 +-
workflows/linux/Kconfig.mcgrof | 2 +
workflows/linux/Kconfig.next | 3 +-
workflows/linux/Kconfig.stable | 3 +-
44 files changed, 852 insertions(+), 63 deletions(-)
create mode 100644 kconfigs/Kconfig.defaults
create mode 100644 playbooks/linux-mirror.yml
create mode 100644 playbooks/roles/linux-mirror/README.md
create mode 100644 playbooks/roles/linux-mirror/defaults/main.yml
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer
create mode 100755 playbooks/roles/linux-mirror/python/gen-mirror-files.py
create mode 100755 playbooks/roles/linux-mirror/python/start-mirroring.py
create mode 100644 playbooks/roles/linux-mirror/tasks/main.yml
create mode 100644 playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
create mode 100755 scripts/append-makefile-vars.sh
create mode 100755 scripts/check_mirror_present.sh
create mode 100755 scripts/get-distro-default-bridge.sh
create mode 100755 scripts/get-distro-prefix.sh
--
2.43.0
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 01/18] linux-mirror: add kdevops mirroring support
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 02/18] Makefile.min_deps: add netcat as a min binary dependency Luis Chamberlain
` (17 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
We use kdevops git trees even on each guest, instead of cloning
on github.com use the local mirror once and keep the mirror up to
date every 5 minutes.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 5 +--
kconfigs/Kconfig.mirror | 32 +++++++++++++++++++
.../linux-mirror-systemd/Makefile | 10 ++++++
.../kdevops-mirror.service | 13 ++++++++
.../linux-mirror-systemd/kdevops-mirror.timer | 10 ++++++
5 files changed, 68 insertions(+), 2 deletions(-)
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index 245aa34cb4ca..762e53562493 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -46,8 +46,8 @@ LOCALHOST_SETUP_WORK += mirror
mirror-help-menu:
@echo "Mirror options:"
- @echo "mirror - sets up systemd mirrors (Linux & QEMU)"
- @echo "mirror-status - checks systemd mirrors status (Linux & QEMU)"
+ @echo "mirror - sets up systemd mirrors"
+ @echo "mirror-status - checks systemd mirrors status"
@echo ""
HELP_TARGETS += mirror-help-menu
@@ -60,3 +60,4 @@ export MCGROF := $(subst ",,$(CONFIG_MIRROR_MCGROF_URL))
export MCGROF_LINUS := $(subst ",,$(CONFIG_MIRROR_MCGROF_LINUS_URL))
export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL))
export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
+export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index 3f0b15859319..0cdce1566a8b 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -34,6 +34,7 @@ config INSTALL_LOCAL_LINUX_MIRROR
to keep these properly updated. The repositories will be placed
into /mirror/ directory:
+ Kernel development trees:
/mirror/linux.git
/mirror/linux-stable.git
/mirror/linux-next.git
@@ -41,6 +42,9 @@ config INSTALL_LOCAL_LINUX_MIRROR
/mirror/mcgrof-next.git
/mirror/mcgrof-linus.git
+ kdevops specific trees and kdevops git alternatives:
+ /mirror/kdevops.git
+
Enabling this even if you already have the repositories does nothing
so it is safe to keep this enabled even after your first linux-mirror
setup. You can however safely also disable this option after your
@@ -51,6 +55,34 @@ config MIRROR_GIT_WORKS
bool
default $(shell, ./scripts/test_git_firewall.sh)
+choice
+ prompt "kdevops mirror source"
+ default MIRROR_KDEVOPS_HTTPS_GITHUB
+ depends on INSTALL_LOCAL_LINUX_MIRROR
+
+config MIRROR_KDEVOPS_HTTPS_GITHUB
+ bool "HTTPS (github)"
+ help
+ If you enable this option then Github HTTPS will be used as the
+ source of the mirror. The URL is:
+
+ https://github.com/linux-kdevops/kdevops.git
+
+config MIRROR_KDEVOPS_HTTPS_GITLAB
+ bool "HTTPS (GitLab)"
+ help
+ If you enable this option then GitLab HTTPS will be used as the
+ source of the mirror. The URL is:
+
+ https://gitlab.com/linux-kdevops/kdevops.git
+
+endchoice
+
+config MIRROR_KDEVOPS_URL
+ string
+ default "https://github.com/linux-kdevops/kdevops.git" if MIRROR_KDEVOPS_HTTPS_GITHUB
+ default "https://gitlab.com/linux-kdevops/kdevops.git" if MIRROR_KDEVOPS_HTTPS_GITLAB
+
choice
prompt "Linux mirror protocol"
default MIRROR_TORVALDS_GIT if MIRROR_GIT_WORKS
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 2bccada39cfc..5568d870c6bc 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -15,6 +15,9 @@ LINUX_TIMERS += kdevops-linus-mirror.timer
LINUX_TIMERS += mcgrof-next-mirror.timer
LINUX_TIMERS += mcgrof-linus-mirror.timer
+KDEVOPS_SERVICES := kdevops-mirror.service
+KDEVOPS_TIMERS := kdevops-mirror.timer
+
QEMU_SERVICES := qemu-mirror.service
QEMU_SERVICES += qemu-jic23-mirror.service
QEMU_TIMERS := qemu-mirror.timer
@@ -32,12 +35,15 @@ MCGROF_TARGET_LINUS := $(MIRROR_PATH)/mcgrof-linus.git
MCGROF_TARGET := $(MIRROR_PATH)/mcgrof-next.git
QEMU_TARGET := $(MIRROR_PATH)/qemu.git
QEMU_JIC23_TARGET := $(MIRROR_PATH)/qemu-jic23.git
+KDEVOPS_TARGET := $(MIRROR_PATH)/kdevops.git
MIRROR_SERVICES := $(LINUX_SERVICES)
MIRROR_SERVICES += $(QEMU_SERVICES)
+MIRROR_SERVICES += $(KDEVOPS_SERVICES)
MIRROR_TIMERS := $(LINUX_TIMERS)
MIRROR_TIMERS += $(QEMU_TIMERS)
+MIRROR_TIMERS += $(KDEVOPS_TIMERS)
ifeq ($(V),1)
export Q=
@@ -96,6 +102,10 @@ mirror:
echo " CLONE qemu-project/jic23-qemu" ;\
git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(QEMU_JIC23) --reference /mirror/qemu.git $(QEMU_JIC23_TARGET) ;\
fi
+ $(Q)if [ ! -d $(KDEVOPS_TARGET) ]; then \
+ echo " CLONE linux-kdevops/kdevops" ;\
+ git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(KDEVOPS) $(KDEVOPS_TARGET) ;\
+ fi
install:
$(Q)mkdir -p $(USER_SYSTEM)
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service
new file mode 100644
index 000000000000..9cc1efdc892b
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=kdevops mirror [qemu.git]
+Documentation=man:git(1)
+ConditionPathExists=/mirror/kdevops.git
+
+[Service]
+Type=oneshot
+ExecStartPre=/usr/bin/git -C /mirror/kdevops.git remote update --prune
+ExecStart=/usr/bin/git -C /mirror/kdevops.git fetch --tags --prune
+ExecStartPost=/usr/bin/git -C /mirror/kdevops.git fetch origin +refs/heads/*:refs/heads/*
+
+[Install]
+WantedBy=multi-user.target
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer
new file mode 100644
index 000000000000..9138bc7494bc
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer
@@ -0,0 +1,10 @@
+[Unit]
+Description=kdevops mirror query timer [kdevops.git]
+ConditionPathExists=/mirror/qemu.git
+
+[Timer]
+OnBootSec=5m
+OnUnitInactiveSec=10m
+
+[Install]
+WantedBy=default.target
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 02/18] Makefile.min_deps: add netcat as a min binary dependency
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
2024-03-30 2:25 ` [PATCH 01/18] linux-mirror: add kdevops mirroring support Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 03/18] provision: generalize bridge-ip further Luis Chamberlain
` (16 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
we use nc to check if your firewall allows git access on the
script ./scripts/test_git_firewall.sh for MIRROR_GIT_WORKS if
you enable ENABLE_LOCAL_LINUX_MIRROR.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.min_deps | 1 +
1 file changed, 1 insertion(+)
diff --git a/Makefile.min_deps b/Makefile.min_deps
index 93d4af6b91e6..9d8f205799c4 100644
--- a/Makefile.min_deps
+++ b/Makefile.min_deps
@@ -8,6 +8,7 @@
BINARY_DEPS :=
BINARY_DEPS += sudo
BINARY_DEPS += make
+BINARY_DEPS += nc
BINARY_DEPS += ansible-playbook
BINARY_DEPS_VCHECKS := $(subst $(TOPDIR)/,,$(wildcard $(TOPDIR)/scripts/version_check/*))
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 03/18] provision: generalize bridge-ip further
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
2024-03-30 2:25 ` [PATCH 01/18] linux-mirror: add kdevops mirroring support Luis Chamberlain
2024-03-30 2:25 ` [PATCH 02/18] Makefile.min_deps: add netcat as a min binary dependency Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 04/18] mirrors: add guestfs mirror options without 9p Luis Chamberlain
` (15 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
We only provide the bridge IP on Kconfig if we're using vagrant or
guestfs, and we wrap some kconfig logic around this. We want to expand
on this and use the default bridge IP for mirroring as well, so that we
don't phone home to the internet when we have a git tree for local
trees. Today we only use git trees for the large trees for the Linux
kernel and qemu but we want to expand on that, and add all of the
mirrors a node fetches to a locall mirror.
To enable more wide spread use of the bridge IP we need to just make
it available on Makefiles, so we can take advantage of the fact that
although kconfig does not let us append variables for example we can't
use "default $(foo)/$(bar)" we *can* as the shell to append scripts for
us, we just need to promote the variables we need to shell.
We provide a simple scripts/append-makefile-vars.sh to make emphasis
that you need these variables on Makefiles so you can use them on
Kconfig.
With the bridge IP for all worlds defined we can now grow its use later
for allowing us to extend how we use bridging using the git protocol on
the local network. We'll address this next.
This makes no functional changes, it just let's us later use the
bridge IP variables on shell Kconfig magic.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
kconfigs/Kconfig.ansible_provisioning | 4 ++--
kconfigs/Kconfig.guestfs | 6 ------
scripts/append-makefile-vars.sh | 10 ++++++++++
scripts/get-distro-default-bridge.sh | 15 +++++++++++++++
scripts/get-distro-prefix.sh | 18 ++++++++++++++++++
scripts/provision.Makefile | 12 ++++++++++++
6 files changed, 57 insertions(+), 8 deletions(-)
create mode 100755 scripts/append-makefile-vars.sh
create mode 100755 scripts/get-distro-default-bridge.sh
create mode 100755 scripts/get-distro-prefix.sh
diff --git a/kconfigs/Kconfig.ansible_provisioning b/kconfigs/Kconfig.ansible_provisioning
index e56f869c16cd..82a6f822c3f5 100644
--- a/kconfigs/Kconfig.ansible_provisioning
+++ b/kconfigs/Kconfig.ansible_provisioning
@@ -194,8 +194,8 @@ config DEVCONFIG_SYSTEMD_JOURNAL_USE_HTTP
config DEVCONFIG_SYSTEMD_JOURNAL_REMOTE_URL
string "systemd-remote URL"
- default "http://192.168.124.1" if VAGRANT
- default GUESTFS_DEFAULT_BRIDGE_IP if GUESTFS
+ default $(shell, scripts/append-makefile-vars.sh http:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT)) if VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh http:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS)) if GUESTFS
help
This is used for the client /etc/systemd/journal-upload.conf URL line.
By default we won't use SSL for now. If you should make sure the
diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
index 377f3434a892..03e0fb865a66 100644
--- a/kconfigs/Kconfig.guestfs
+++ b/kconfigs/Kconfig.guestfs
@@ -1,11 +1,5 @@
if GUESTFS
-config GUESTFS_DEFAULT_BRIDGE_IP
- string
- default "http://192.168.122.1" if DISTRO_FEDORA || DISTRO_REDHAT
- default "http://192.168.122.1" if DISTRO_OPENSUSE || DISTRO_SUSE
- default "http://192.168.122.1" if DISTRO_DEBIAN || DISTRO_UBUNTU
-
choice
prompt "Guestfs Linux distribution to use"
default GUESTFS_FEDORA if DISTRO_FEDORA || DISTRO_REDHAT
diff --git a/scripts/append-makefile-vars.sh b/scripts/append-makefile-vars.sh
new file mode 100755
index 000000000000..2148399ae31f
--- /dev/null
+++ b/scripts/append-makefile-vars.sh
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+STR=""
+
+while [[ ${#1} -gt 0 ]]; do
+ STR="${STR}${1}"
+ shift
+done
+
+echo $STR
diff --git a/scripts/get-distro-default-bridge.sh b/scripts/get-distro-default-bridge.sh
new file mode 100755
index 000000000000..09d09ff04ee1
--- /dev/null
+++ b/scripts/get-distro-default-bridge.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+DISTRO=$1
+VIRT_TYPE=$2
+
+# We currently ignore the distro because as far as we're concerned
+# they all use the default IP, the only thing that differiantes it
+# is if you are using vagrant or guestfs. If your distribution needs
+# a different deafult it should be easy for you to extend it here.
+
+if [[ "$VIRT_TYPE" == "vagrant" ]]; then
+ echo "http://192.168.124.1"
+else
+ echo "192.168.122.1"
+fi
diff --git a/scripts/get-distro-prefix.sh b/scripts/get-distro-prefix.sh
new file mode 100755
index 000000000000..519be5a5a97c
--- /dev/null
+++ b/scripts/get-distro-prefix.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+DEFAULT_DISTRO="debian"
+DISTROS="debian"
+DISTROS="$DISTROS fedora"
+DISTROS="$DISTROS opensuse"
+DISTROS="$DISTROS redhat"
+DISTROS="$DISTROS suse"
+DISTROS="$DISTROS ubuntu"
+
+for i in $DISTROS; do
+ if [[ "./scripts/os-release-check.sh $i" == "y" ]]; then
+ echo $i
+ exit
+ fi
+done
+
+echo $DEFAULT_DISTRO
diff --git a/scripts/provision.Makefile b/scripts/provision.Makefile
index 7bd4115bdb7a..26657c0974f2 100644
--- a/scripts/provision.Makefile
+++ b/scripts/provision.Makefile
@@ -31,6 +31,18 @@ KDEVOPS_PROVISIONED_DEVCONFIG := .provisioned_once_devconfig
KDEVOPS_BRING_UP_DEPS :=
KDEVOPS_DESTROY_DEPS :=
+# This is currently used when vitualization is enabled so to allow us to
+# allow guests to use the bridge ip for mirroring git trees but the general idea
+# could be extended to support cloud later. For example perhaps this could be
+# a third guest which we can instantiate later. This would be useful for things
+# like systemd-remote as well.
+#
+# We export this into Makefiles so that we can use them both on Makefiles and
+# on Kconfig files with environment variables such as $(FOO)
+export KDEVOPS_DEFAULT_DISTRO := $(shell $(TOPDIR)/scripts/get-distro-prefix.sh)
+export KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT := $(shell $(TOPDIR)/scripts/get-distro-default-bridge.sh $(KDEVOPS_DEFAULT_DISTRO) vagrant)
+export KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS := $(shell $(TOPDIR)/scripts/get-distro-default-bridge.sh $(KDEVOPS_DEFAULT_DISTRO) guestfs)
+
# These go last
KDEVOPS_BRING_UP_LATE_DEPS :=
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 04/18] mirrors: add guestfs mirror options without 9p
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (2 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 03/18] provision: generalize bridge-ip further Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 05/18] mirrors: enable kdevops mirror Luis Chamberlain
` (14 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
When 9p is disabled git cloning using mirroring needs to rely on the
bridge IP which is different on guestfs than on vagrant. Add the proper
bridge IP this case.
Note that most users use 9p, so this just fixes the mirroring option
when 9p is disabled.
signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
workflows/linux/Kconfig.linus | 3 ++-
workflows/linux/Kconfig.mcgrof | 2 ++
workflows/linux/Kconfig.next | 3 ++-
workflows/linux/Kconfig.stable | 3 ++-
4 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/workflows/linux/Kconfig.linus b/workflows/linux/Kconfig.linus
index 5c524e5266ae..14d3e0c0f360 100644
--- a/workflows/linux/Kconfig.linus
+++ b/workflows/linux/Kconfig.linus
@@ -69,7 +69,8 @@ config BOOTLINUX_TREE_LINUS_NAME
config BOOTLINUX_TREE_LINUS_URL
string
default "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" if !USE_LOCAL_LINUX_MIRROR
- default "git://192.168.124.1/mirror/linux.git" if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/linux.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/linux.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && GUESTFS
default "/mirror/linux.git" if USE_LIBVIRT_MIRROR && BOOTLINUX_9P
endif # BOOTLINUX_TREE_LINUS
diff --git a/workflows/linux/Kconfig.mcgrof b/workflows/linux/Kconfig.mcgrof
index 4c4be5b72aff..f2d2f898234f 100644
--- a/workflows/linux/Kconfig.mcgrof
+++ b/workflows/linux/Kconfig.mcgrof
@@ -65,5 +65,7 @@ config BOOTLINUX_MCGROF_TREE_URL
default "/mirror/mcgrof-next.git" if BOOTLINUX_TREE_MCGROF_NEXT && USE_LIBVIRT_MIRROR && BOOTLINUX_9P
default "https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git" if BOOTLINUX_TREE_MCGROF_LINUS && !USE_LOCAL_LINUX_MIRROR
default "/mirror/mcgrof-linus.git" if BOOTLINUX_TREE_MCGROF_LINUS && USE_LIBVIRT_MIRROR && BOOTLINUX_9P
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/mcgrof-linus.git) if BOOTLINUX_TREE_MCGROF_LINUS && USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/mcgrof-next.git) if BOOTLINUX_TREE_MCGROF_NEXT && USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && GUESTFS
endif # BOOTLINUX_TREE_MCGROF_LINUS || BOOTLINUX_TREE_MCGROF_NEXT
diff --git a/workflows/linux/Kconfig.next b/workflows/linux/Kconfig.next
index 48c86e3dbc08..6fdb53077aa7 100644
--- a/workflows/linux/Kconfig.next
+++ b/workflows/linux/Kconfig.next
@@ -11,7 +11,8 @@ config BOOTLINUX_TREE_NEXT_NAME
config BOOTLINUX_TREE_NEXT_URL
string
default "https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git" if !USE_LOCAL_LINUX_MIRROR
- default "git://192.168.124.1/mirror/linux-next.git" if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/linux-next.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/linux-next.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && GUESTFS
default "/mirror/linux-next.git" if USE_LIBVIRT_MIRROR && BOOTLINUX_9P
endif # BOOTLINUX_TREE_NEXT
diff --git a/workflows/linux/Kconfig.stable b/workflows/linux/Kconfig.stable
index 39c1984a0630..a0eaa1bffadf 100644
--- a/workflows/linux/Kconfig.stable
+++ b/workflows/linux/Kconfig.stable
@@ -85,7 +85,8 @@ config BOOTLINUX_TREE_STABLE_NAME
config BOOTLINUX_TREE_STABLE_URL
string
default "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git" if !USE_LOCAL_LINUX_MIRROR
- default "git://192.168.124.1/mirror/linux-stable.git" if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/linux-stable.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/linux-stable.git) if USE_LIBVIRT_MIRROR && !BOOTLINUX_9P && GUESTFS
default "/mirror/linux-stable.git" if USE_LIBVIRT_MIRROR && BOOTLINUX_9P
endif # BOOTLINUX_STABLE
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 05/18] mirrors: enable kdevops mirror
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (3 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 04/18] mirrors: add guestfs mirror options without 9p Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 06/18] mirrors: add fstests to mirrors Luis Chamberlain
` (13 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
We recently added support for mirroring kdevops but didn't enable it
yet as we needed some more work, the work to get the bridge IP in a
clean way. Now that previous commits did the work for that enable
the mirror to be used, but only if we detect that the local server
has a copy of that mirror. The reason we check for the mirror is
that prior mirroring attempts may have been made with older versions
of kdevops and they would not have set up the kdevops mirror yet.
We therefore only enable the kdevops mirror to be used if and only if
the localhost has it present.
This is going to save us tons of bandiwdth, as each guest when testing
filesystems was git cloning kdevops and kdevops already is about over
200 MiB.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
kconfigs/workflows/Kconfig.shared | 11 +++++++++--
scripts/check_mirror_present.sh | 11 +++++++++++
2 files changed, 20 insertions(+), 2 deletions(-)
create mode 100755 scripts/check_mirror_present.sh
diff --git a/kconfigs/workflows/Kconfig.shared b/kconfigs/workflows/Kconfig.shared
index c4aab329e7b1..029eaa4383db 100644
--- a/kconfigs/workflows/Kconfig.shared
+++ b/kconfigs/workflows/Kconfig.shared
@@ -75,11 +75,18 @@ endif # WORKFLOW_KDEVOPS_GIT_CUSTOM
if !WORKFLOW_KDEVOPS_GIT_CUSTOM
+config HAVE_MIRROR_KDEVOPS
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/kdevops.git)
+
config WORKFLOW_KDEVOPS_GIT
string
default "https://github.com/linux-kdevops/kdevops.git" if !GIT_ALTERNATIVES && !HAVE_CUSTOM_KDEVOPS_GIT
- default "https://github.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITHUB
- default "https://gitlab.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITLAB
+ default "https://github.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITHUB && !HAVE_MIRROR_KDEVOPS
+ default "https://gitlab.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITLAB && !HAVE_MIRROR_KDEVOPS
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/kdevops.git) if HAVE_MIRROR_KDEVOPS && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/kdevops.git) if HAVE_MIRROR_KDEVOPS && GUESTFS
help
Some workflows want to also clone a version of kdevops on each target.
An example of this is when using the fstests workflow, which carries
diff --git a/scripts/check_mirror_present.sh b/scripts/check_mirror_present.sh
new file mode 100755
index 000000000000..b99c7048c12e
--- /dev/null
+++ b/scripts/check_mirror_present.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+MIRROR_PATH="$1"
+
+if [[ -d $MIRROR_PATH ]]; then
+ echo y
+ exit
+fi
+
+echo n
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 06/18] mirrors: add fstests to mirrors
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (4 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 05/18] mirrors: enable kdevops mirror Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 07/18] fstests: use local primary fstests mirror if present Luis Chamberlain
` (12 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
The fstests workflow relies on fstests, so mirror that as well.
This is the first general tool we are mirroring, we will make
of use of it next in the subsequent patch. We update this tree
every 6 hours.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 1 +
kconfigs/Kconfig.mirror | 44 ++++++++++++++++++-
.../linux-mirror-systemd/Makefile | 10 +++++
.../fstests-mirror.service | 13 ++++++
.../linux-mirror-systemd/fstests-mirror.timer | 11 +++++
5 files changed, 78 insertions(+), 1 deletion(-)
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index 762e53562493..d6b0c848eaeb 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -61,3 +61,4 @@ export MCGROF_LINUS := $(subst ",,$(CONFIG_MIRROR_MCGROF_LINUS_URL))
export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL))
export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
+export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL))
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index 0cdce1566a8b..fa5ec8c1333f 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -45,6 +45,9 @@ config INSTALL_LOCAL_LINUX_MIRROR
kdevops specific trees and kdevops git alternatives:
/mirror/kdevops.git
+ Tools:
+ /mirror/fstests.git
+
Enabling this even if you already have the repositories does nothing
so it is safe to keep this enabled even after your first linux-mirror
setup. You can however safely also disable this option after your
@@ -387,5 +390,44 @@ config MIRROR_QEMU_JIC23_URL
default "https://gitlab.com/jic23/qemu.git" if MIRROR_QEMU_JIC23_HTTPS_GITLAB
default "ssh://git@gitlab.com:jic23/qemu.git" if MIRROR_QEMU_JIC23_SSH_GITLAB
-endif
+choice
+ prompt "Tools - xfstests-dev.git mirror protocol/source"
+ default MIRROR_FSTESTS_GIT if MIRROR_GIT_WORKS
+ default MIRROR_FSTESTS_HTTPS if !MIRROR_GIT_WORKS
+ depends on INSTALL_LOCAL_LINUX_MIRROR
+
+config MIRROR_FSTESTS_GIT
+ bool "Git"
+ depends on MIRROR_GIT_WORKS
+ help
+ If you enable this option then git:// protocol will be used as the
+ source of the mirror. The URL is:
+
+ git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
+
+config MIRROR_FSTESTS_HTTPS
+ bool "HTTPS"
+ help
+ If you enable this option then GitLab HTTPS will be used as the
+ source of the mirror. The URL is:
+
+ https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
+
+config MIRROR_FSTESTS_HTTPS_GOOGLE
+ bool "HTTPS (Google)"
+ help
+ If you enable this option then Google HTTPS will be used as the source
+ of the mirror. The URL is:
+
+ https://kernel.googlesource.com/pub/scm/fs/xfs/xfstests-dev.git
+
+endchoice
+
+config MIRROR_FSTESTS_URL
+ string
+ default "git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_GIT
+ default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS
+ default "https://kernel.googlesource.com/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS_GOOGLE
+
+endif # ENABLE_LOCAL_LINUX_MIRROR
endif # TERRAFORM
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 5568d870c6bc..7de8cc4e326d 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -18,6 +18,9 @@ LINUX_TIMERS += mcgrof-linus-mirror.timer
KDEVOPS_SERVICES := kdevops-mirror.service
KDEVOPS_TIMERS := kdevops-mirror.timer
+TOOLS_SERVICES := fstests-mirror.service
+TOOLS_TIMERS := fstests-mirror.timer
+
QEMU_SERVICES := qemu-mirror.service
QEMU_SERVICES += qemu-jic23-mirror.service
QEMU_TIMERS := qemu-mirror.timer
@@ -36,14 +39,17 @@ MCGROF_TARGET := $(MIRROR_PATH)/mcgrof-next.git
QEMU_TARGET := $(MIRROR_PATH)/qemu.git
QEMU_JIC23_TARGET := $(MIRROR_PATH)/qemu-jic23.git
KDEVOPS_TARGET := $(MIRROR_PATH)/kdevops.git
+FSTESTS_TARGET := $(MIRROR_PATH)/fstests.git
MIRROR_SERVICES := $(LINUX_SERVICES)
MIRROR_SERVICES += $(QEMU_SERVICES)
MIRROR_SERVICES += $(KDEVOPS_SERVICES)
+MIRROR_SERVICES += $(TOOLS_SERVICES)
MIRROR_TIMERS := $(LINUX_TIMERS)
MIRROR_TIMERS += $(QEMU_TIMERS)
MIRROR_TIMERS += $(KDEVOPS_TIMERS)
+MIRROR_TIMERS += $(TOOLS_TIMERS)
ifeq ($(V),1)
export Q=
@@ -106,6 +112,10 @@ mirror:
echo " CLONE linux-kdevops/kdevops" ;\
git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(KDEVOPS) $(KDEVOPS_TARGET) ;\
fi
+ $(Q)if [ ! -d $(FSTESTS_TARGET) ]; then \
+ echo " CLONE fstests" ;\
+ git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(FSTESTS) $(FSTESTS_TARGET) ;\
+ fi
install:
$(Q)mkdir -p $(USER_SYSTEM)
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service b/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service
new file mode 100644
index 000000000000..31ce3e7552e9
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=fstests mirror [fstests.git]
+Documentation=man:git(1)
+ConditionPathExists=/mirror/fstests.git
+
+[Service]
+Type=oneshot
+ExecStartPre=/usr/bin/git -C /mirror/fstests.git remote update --prune
+ExecStart=/usr/bin/git -C /mirror/fstests.git fetch --tags --prune
+ExecStartPost=/usr/bin/git -C /mirror/fstests.git fetch origin +refs/heads/*:refs/heads/*
+
+[Install]
+WantedBy=multi-user.target
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer b/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer
new file mode 100644
index 000000000000..0a69f2f55897
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=fstests mirror query timer [fstests.git]
+ConditionPathExists=/mirror/fstests.git
+
+[Timer]
+OnBootSec=10m
+# Every 6 hours
+OnUnitInactiveSec=360m
+
+[Install]
+WantedBy=default.target
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 07/18] fstests: use local primary fstests mirror if present
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (5 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 06/18] mirrors: add fstests to mirrors Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 08/18] mirrors: add kdevops git alternative for fstests Luis Chamberlain
` (11 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
There are two ways to use tools:
a) primary upstream
b) alternatives
The b) alternatives were implemented in kdeovps using
CONFIG_GIT_ALTERNATIVES so to allow us to enable reproducible
build environments, letting you keep forks of the primary source
tree.
In practice for fstest that means kdevops has its own fstests
git tree under the kdevops organization, in case that is desirable.
That is actually the default, so to avoid issues with using the
latest upstream fstests tree.
This change is when you use a) the primary upstream tree.
So when you disable CONFIG_GIT_ALTERNATIVES, we now end up using
the local mirror if you have it present:
grep fstests_git extra_vars.yaml
fstests_git: git://192.168.122.1/mirror/fstests.git
Note that today's default is with CONFIG_GIT_ALTERNATIVES enabled.
We'll be addressing this later after we first add mirroring
support for kdevops's own fork of fstests.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
workflows/fstests/Kconfig | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
index d65df7f0ecfa..4b22be71c805 100644
--- a/workflows/fstests/Kconfig
+++ b/workflows/fstests/Kconfig
@@ -275,9 +275,16 @@ endmenu
endif # FSTESTS_TMPFS
+config HAVE_MIRROR_FSTESTS
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/fstests.git)
+
config FSTESTS_GIT
string "The fstests git tree to clone"
- default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if !GIT_ALTERNATIVES
+ default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if !GIT_ALTERNATIVES && !HAVE_MIRROR_FSTESTS
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/fstests.git) if !GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/fstests.git) if !GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && GUESTFS
default "https://github.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITHUB
default "https://gitlab.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITLAB
default "https://github.com/btrfs/fstests.git" if FSTESTS_BTRFS
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 08/18] mirrors: add kdevops git alternative for fstests
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (6 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 07/18] fstests: use local primary fstests mirror if present Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 09/18] mirrors: use kdevops fstests mirror when available Luis Chamberlain
` (10 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
kdevops supports its own CONFIG_GIT_ALTERNATIVES which allows
kdevops to mirror upstream projects on its own organization, so
that our own fork can host patches which perhaps have not landed
upstream on fstests yet, and also to ensure reproducible workflows.
If we follow upstream, the branch may be a moving target and sometimes
thigns may work sometimes things will not. For deatils refer to our
Kconfig documentation of GIT_ALTERNATIVES.
This now mirrors the fstests kdevops git alternatives. We will place
these mirrored alternatives under a directory representing the
alternative source, in this case its kdevops, /mirror/kdevops/fstest.git.
Note that we have an alternative for fstests for btrfs but this doesn't
yet properly pronounce itself as an alternative itself, its was just
caked in as another option. We should fix that so that we can also
mirror btrfs's fstests and be able to easily use it.
Now with today's defaults you will end up using mirroring for
both kdevops and fstests when you enable the fstests workflow
if a mirror was found for each one on your system:
grep git extra_vars.yaml | grep git
kdevops_git: git://192.168.122.1/mirror/kdevops.git
kdevops_git_version: HEAD
fstests_git: git://192.168.122.1/mirror/kdevops/fstests.git
fstests_git_version: HEAD
It is important to use --reference here as the kdevops fork and so
can share objects on the server with the fstests tree. If you add
support for a tree which is a fork, do the same.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 1 +
kconfigs/Kconfig.mirror | 30 +++++++++++++++++++
.../linux-mirror-systemd/Makefile | 7 +++++
.../kdevops-fstests-mirror.service | 13 ++++++++
.../kdevops-fstests-mirror.timer | 11 +++++++
5 files changed, 62 insertions(+)
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index d6b0c848eaeb..f6cc1b447eb8 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -62,3 +62,4 @@ export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL))
export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL))
+export KDEVOPS_FSTESTS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_FSTESTS_URL))
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index fa5ec8c1333f..076d5d117984 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -44,6 +44,7 @@ config INSTALL_LOCAL_LINUX_MIRROR
kdevops specific trees and kdevops git alternatives:
/mirror/kdevops.git
+ /mirror/kdevops/fstests.git
Tools:
/mirror/fstests.git
@@ -429,5 +430,34 @@ config MIRROR_FSTESTS_URL
default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS
default "https://kernel.googlesource.com/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS_GOOGLE
+choice
+ prompt "Tools - kdevops/fstests.git mirror protocol/source"
+ default MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB
+ default MIRROR_KDEOVPS_FSTESTS_HTTPS_GITLAB
+ depends on INSTALL_LOCAL_LINUX_MIRROR
+
+config MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB
+ bool "HTTPS (GitHub)"
+ help
+ If you enable this option then GitHub HTTPS will be used as the
+ source of the mirror. The URL is:
+
+ https://github.com/linux-kdevops/fstests.git
+
+config MIRROR_KDEVOPS_FSTESTS_HTTPS_GITLAB
+ bool "HTTPS (GitLab)"
+ help
+ If you enable this option then GitLab HTTPS will be used as the
+ source of the mirror. The URL is:
+
+ https://gitlab.com/linux-kdevops/fstests.git
+
+endchoice
+
+config MIRROR_KDEVOPS_FSTESTS_URL
+ string
+ default "https://github.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB
+ default "https://gitlab.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITLAB
+
endif # ENABLE_LOCAL_LINUX_MIRROR
endif # TERRAFORM
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 7de8cc4e326d..4a69ee7ca88f 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -19,7 +19,9 @@ KDEVOPS_SERVICES := kdevops-mirror.service
KDEVOPS_TIMERS := kdevops-mirror.timer
TOOLS_SERVICES := fstests-mirror.service
+TOOLS_SERVICES += kdevops-fstests-mirror.service
TOOLS_TIMERS := fstests-mirror.timer
+TOOLS_TIMERS += kdevops-fstests-mirror.timer
QEMU_SERVICES := qemu-mirror.service
QEMU_SERVICES += qemu-jic23-mirror.service
@@ -40,6 +42,7 @@ QEMU_TARGET := $(MIRROR_PATH)/qemu.git
QEMU_JIC23_TARGET := $(MIRROR_PATH)/qemu-jic23.git
KDEVOPS_TARGET := $(MIRROR_PATH)/kdevops.git
FSTESTS_TARGET := $(MIRROR_PATH)/fstests.git
+KDEVOPS_FSTESTS_TARGET := $(MIRROR_PATH)/kdevops/fstests.git
MIRROR_SERVICES := $(LINUX_SERVICES)
MIRROR_SERVICES += $(QEMU_SERVICES)
@@ -116,6 +119,10 @@ mirror:
echo " CLONE fstests" ;\
git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(FSTESTS) $(FSTESTS_TARGET) ;\
fi
+ $(Q)if [ ! -d $(KDEVOPS_FSTESTS_TARGET) ]; then \
+ echo " CLONE kdevops-fstests" ;\
+ git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(KDEVOPS_FSTESTS) --reference $(FSTESTS_TARGET) $(KDEVOPS_FSTESTS_TARGET) ;\
+ fi
install:
$(Q)mkdir -p $(USER_SYSTEM)
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service
new file mode 100644
index 000000000000..822a62548937
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=kdevops-fstests mirror [kdevops/fstests.git]
+Documentation=man:git(1)
+ConditionPathExists=/mirror/kdevops/fstests.git
+
+[Service]
+Type=oneshot
+ExecStartPre=/usr/bin/git -C /mirror/kdevops/fstests.git remote update --prune
+ExecStart=/usr/bin/git -C /mirror/kdevops/fstests.git fetch --tags --prune
+ExecStartPost=/usr/bin/git -C /mirror/kdevops/fstests.git fetch origin +refs/heads/*:refs/heads/*
+
+[Install]
+WantedBy=multi-user.target
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer
new file mode 100644
index 000000000000..4ec7a794b488
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=kdevops fstests mirror query timer [kdevops/fstests.git]
+ConditionPathExists=/mirror/kdevops/fstests.git
+
+[Timer]
+OnBootSec=10m
+# Every 6 hours
+OnUnitInactiveSec=360m
+
+[Install]
+WantedBy=default.target
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 09/18] mirrors: use kdevops fstests mirror when available
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (7 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 08/18] mirrors: add kdevops git alternative for fstests Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 10/18] mirrors: add blktests mirror Luis Chamberlain
` (9 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
This now extends support to use the kdevops specific mirror of fstests
if its present.
Now, with today's defaults you will end up using mirroring for
both kdevops and fstests when you enable the fstests workflow
if a mirror was found for each one on your system:
grep git extra_vars.yaml | grep git
kdevops_git: git://192.168.122.1/mirror/kdevops.git
kdevops_git_version: HEAD
fstests_git: git://192.168.122.1/mirror/kdevops/fstests.git
fstests_git_version: HEAD
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
workflows/fstests/Kconfig | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
index 4b22be71c805..c797b2491d5a 100644
--- a/workflows/fstests/Kconfig
+++ b/workflows/fstests/Kconfig
@@ -280,13 +280,20 @@ config HAVE_MIRROR_FSTESTS
depends on USE_LIBVIRT_MIRROR
default $(shell, scripts/check_mirror_present.sh /mirror/fstests.git)
+config HAVE_MIRROR_KDEVOPS_FSTESTS
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/kdevops/fstests.git)
+
config FSTESTS_GIT
string "The fstests git tree to clone"
default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if !GIT_ALTERNATIVES && !HAVE_MIRROR_FSTESTS
default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/fstests.git) if !GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && VAGRANT
default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/fstests.git) if !GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && GUESTFS
- default "https://github.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITHUB
- default "https://gitlab.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITLAB
+ default "https://github.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITHUB && !HAVE_MIRROR_KDEVOPS_FSTESTS
+ default "https://gitlab.com/linux-kdevops/fstests.git" if GIT_LINUX_KDEVOPS_GITLAB && !HAVE_MIRROR_KDEVOPS_FSTESTS
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/kdevops/fstests.git) if GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/kdevops/fstests.git) if GIT_ALTERNATIVES && HAVE_MIRROR_FSTESTS && GUESTFS
default "https://github.com/btrfs/fstests.git" if FSTESTS_BTRFS
help
The fstests git tree to clone.
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 10/18] mirrors: add blktests mirror
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (8 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 09/18] mirrors: use kdevops fstests mirror when available Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 11/18] mirrors: use blktests mirror when its available Luis Chamberlain
` (8 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
This adds mirroring for blktests. Only Omar's tree exists, which
means a new mirror is desirable in the future.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 1 +
kconfigs/Kconfig.mirror | 5 +++++
.../linux-mirror/linux-mirror-systemd/Makefile | 7 +++++++
.../linux-mirror-systemd/blktests-mirror.service | 13 +++++++++++++
.../linux-mirror-systemd/blktests-mirror.timer | 11 +++++++++++
5 files changed, 37 insertions(+)
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service
create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index f6cc1b447eb8..ebaf505a7627 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -63,3 +63,4 @@ export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL))
export KDEVOPS_FSTESTS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_FSTESTS_URL))
+export BLKTESTS := $(subst ",,$(CONFIG_MIRROR_BLKTESTS_URL))
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index 076d5d117984..7e17d59a605c 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -48,6 +48,7 @@ config INSTALL_LOCAL_LINUX_MIRROR
Tools:
/mirror/fstests.git
+ /mirror/blktests.git
Enabling this even if you already have the repositories does nothing
so it is safe to keep this enabled even after your first linux-mirror
@@ -459,5 +460,9 @@ config MIRROR_KDEVOPS_FSTESTS_URL
default "https://github.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB
default "https://gitlab.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITLAB
+config MIRROR_BLKTESTS_URL
+ string
+ default "https://github.com/osandov/blktests.git"
+
endif # ENABLE_LOCAL_LINUX_MIRROR
endif # TERRAFORM
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 4a69ee7ca88f..9e03ebd04c97 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -20,8 +20,10 @@ KDEVOPS_TIMERS := kdevops-mirror.timer
TOOLS_SERVICES := fstests-mirror.service
TOOLS_SERVICES += kdevops-fstests-mirror.service
+TOOLS_SERVICES += blktests-mirror.service
TOOLS_TIMERS := fstests-mirror.timer
TOOLS_TIMERS += kdevops-fstests-mirror.timer
+TOOLS_TIMERS += blktests-mirror.timer
QEMU_SERVICES := qemu-mirror.service
QEMU_SERVICES += qemu-jic23-mirror.service
@@ -43,6 +45,7 @@ QEMU_JIC23_TARGET := $(MIRROR_PATH)/qemu-jic23.git
KDEVOPS_TARGET := $(MIRROR_PATH)/kdevops.git
FSTESTS_TARGET := $(MIRROR_PATH)/fstests.git
KDEVOPS_FSTESTS_TARGET := $(MIRROR_PATH)/kdevops/fstests.git
+BLKTESTS_TARGET := $(MIRROR_PATH)/blktests.git
MIRROR_SERVICES := $(LINUX_SERVICES)
MIRROR_SERVICES += $(QEMU_SERVICES)
@@ -123,6 +126,10 @@ mirror:
echo " CLONE kdevops-fstests" ;\
git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(KDEVOPS_FSTESTS) --reference $(FSTESTS_TARGET) $(KDEVOPS_FSTESTS_TARGET) ;\
fi
+ $(Q)if [ ! -d $(BLKTESTS_TARGET) ]; then \
+ echo " CLONE blktests" ;\
+ git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(BLKTESTS) $(BLKTESTS_TARGET) ;\
+ fi
install:
$(Q)mkdir -p $(USER_SYSTEM)
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service b/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service
new file mode 100644
index 000000000000..9e1f82ed32db
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=blktests mirror [blktest.git]
+Documentation=man:git(1)
+ConditionPathExists=/mirror/blktests.git
+
+[Service]
+Type=oneshot
+ExecStartPre=/usr/bin/git -C /mirror/blktests.git remote update --prune
+ExecStart=/usr/bin/git -C /mirror/blktests.git fetch --tags --prune
+ExecStartPost=/usr/bin/git -C /mirror/blktests.git fetch origin +refs/heads/*:refs/heads/*
+
+[Install]
+WantedBy=multi-user.target
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer b/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer
new file mode 100644
index 000000000000..ef2bfb1ecdba
--- /dev/null
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=blktests mirror query timer blktests.git]
+ConditionPathExists=/mirror/blktests.git
+
+[Timer]
+OnBootSec=10m
+# Every 6 hours
+OnUnitInactiveSec=360m
+
+[Install]
+WantedBy=default.target
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 11/18] mirrors: use blktests mirror when its available
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (9 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 10/18] mirrors: add blktests mirror Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 12/18] mirrors: increase scope of mirroring in output Luis Chamberlain
` (7 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
There isn't any blktests mirror out there yet, and we also don't even
have a form of it on kdevops since it is a rather slow moving target
compared to fstests and does not break as often. So mirroring for it
is pretty straight forward, and the only difference is the IP address
betwen the bridge for vagrant and guestfs. Enable the mirror if the
localhost is detected to use mirroring.
After enabling the blktests workflow and running make we end up
with:
grep blktests_git extra_vars.yaml
blktests_git: git://192.168.122.1/mirror/blktests.git
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
workflows/blktests/Kconfig | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/workflows/blktests/Kconfig b/workflows/blktests/Kconfig
index ed0057378ba7..2be13de90a76 100644
--- a/workflows/blktests/Kconfig
+++ b/workflows/blktests/Kconfig
@@ -127,9 +127,16 @@ config BLKTESTS_WATCHDOG_RESET_HUNG_SYSTEMS
endif # BLKTESTS_WATCHDOG
+config HAVE_MIRROR_BLKTESTS
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/blktests.git)
+
config BLKTESTS_GIT
string "The blktests git tree to clone"
- default "https://github.com/osandov/blktests.git"
+ default "https://github.com/osandov/blktests.git" if !HAVE_MIRROR_BLKTESTS
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/blktests.git) if HAVE_MIRROR_BLKTESTS && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/blktests.git) if HAVE_MIRROR_BLKTESTS && GUESTFS
help
The blktests git tree to clone.
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 12/18] mirrors: increase scope of mirroring in output
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (10 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 11/18] mirrors: use blktests mirror when its available Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 13/18] mirrors: move status check Luis Chamberlain
` (6 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Clarify that we are mirroring more than just Linux and qemu now.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 2 +-
playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index ebaf505a7627..3968acecb98e 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -19,7 +19,7 @@ MIRROR_CODE := $(TOPDIR)/playbooks/roles/linux-mirror/linux-mirror-systemd/
# Since we don't support varabiliity yet, for now we keep it very simple:
mirror: $(KDEVOPS_EXTRA_VARS)
- $(NQ) Setting up Linux and QEMU systemd mirrors on localhost
+ $(Q)echo Setting up all mirrors on localhost
$(Q)make -C $(MIRROR_CODE) mirror
$(Q)make -C $(MIRROR_CODE) install
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 9e03ebd04c97..b773c0a2e70a 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -72,7 +72,7 @@ endif
all: help
help:
- $(NQ) "mirror: git clone all mirrors about Linux and QEMU"
+ $(NQ) "mirror: git clone all mirrors"
$(NQ) "install: install system timers and git-daemon socket activation"
mirror:
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 13/18] mirrors: move status check
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (11 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 12/18] mirrors: increase scope of mirroring in output Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 14/18] mirrors: move mirror editing into one file Luis Chamberlain
` (5 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Move most of the 'make mirror-status' work onto the makefile
which already has most of the timer and services already so
we can just use the variables there.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 13 +------------
.../linux-mirror/linux-mirror-systemd/Makefile | 8 ++++++++
2 files changed, 9 insertions(+), 12 deletions(-)
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index 3968acecb98e..40532eb0e477 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -26,18 +26,7 @@ mirror: $(KDEVOPS_EXTRA_VARS)
PHONY += mirror
mirror-status: $(KDEVOPS_EXTRA_VARS)
- $(NQ) systemd timer units list
- $(Q)systemctl --user list-timers
- $(NQ) systemd timer status
- $(Q)systemctl status --user linux-mirror.timer linux-stable-mirror.timer \
- linux-next-mirror.timer mcgrof-next-mirror.timer \
- kdevops-linus-mirror.timer \
- qemu-mirror.timer jic23-qemu-mirror.timer
- $(NQ) systemd service units status
- $(Q)systemctl status --user linux-mirror.service linux-stable-mirror.service \
- kdevops-linus-mirror.service \
- linux-next-mirror.service mcgrof-next-mirror.service \
- qemu-mirror.service jic23-qemu-mirror.timer
+ $(Q)make -C $(MIRROR_CODE) status
PHONY += mirror-status
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index b773c0a2e70a..367ffdc526df 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -149,3 +149,11 @@ install:
$(Q)sudo cp $(GIT_DAEMON_FILES) $(LOCAL_SYSTEMD)
$(Q)sudo systemctl enable git-daemon.socket
$(Q)sudo systemctl start git-daemon.socket
+
+status:
+ $(NQ) systemd timer units list
+ $(Q)systemctl --user list-timers
+ $(NQ) systemd timer status
+ $(Q)systemctl status --user $(MIRROR_TIMERS)
+ $(NQ) systemd service units status
+ $(Q)systemctl status --user $(MIRROR_SERVICES)
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 14/18] mirrors: move mirror editing into one file
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (12 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 13/18] mirrors: move status check Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 15/18] mirrors: add new mirrors.yaml and add xfsprogs Luis Chamberlain
` (4 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
When we add a mirror we edit two Makefiles, reduce this to just one.
To do this we must ensure we can include .config on it, use the full
path to make that work.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 13 -------------
.../linux-mirror/linux-mirror-systemd/Makefile | 18 ++++++++++++++++++
2 files changed, 18 insertions(+), 13 deletions(-)
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index 40532eb0e477..c355c4a0ea98 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -40,16 +40,3 @@ mirror-help-menu:
@echo ""
HELP_TARGETS += mirror-help-menu
-
-export TORVALDS := $(subst ",,$(CONFIG_MIRROR_TORVALDS_URL))
-export STABLE := $(subst ",,$(CONFIG_MIRROR_STABLE_URL))
-export NEXT := $(subst ",,$(CONFIG_MIRROR_NEXT_URL))
-export KDEVOPS_LINUS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_LINUS_URL))
-export MCGROF := $(subst ",,$(CONFIG_MIRROR_MCGROF_URL))
-export MCGROF_LINUS := $(subst ",,$(CONFIG_MIRROR_MCGROF_LINUS_URL))
-export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL))
-export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
-export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
-export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL))
-export KDEVOPS_FSTESTS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_FSTESTS_URL))
-export BLKTESTS := $(subst ",,$(CONFIG_MIRROR_BLKTESTS_URL))
diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
index 367ffdc526df..61b0c5195bc9 100644
--- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
+++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile
@@ -1,6 +1,24 @@
USER_SYSTEM := ~/.config/systemd/user/
MIRROR_PATH := /mirror
+ifeq (,$(wildcard $(TOPDIR_PATH)/.config))
+else
+include $(TOPDIR_PATH)/.config
+endif
+
+export TORVALDS := $(subst ",,$(CONFIG_MIRROR_TORVALDS_URL))
+export STABLE := $(subst ",,$(CONFIG_MIRROR_STABLE_URL))
+export NEXT := $(subst ",,$(CONFIG_MIRROR_NEXT_URL))
+export KDEVOPS_LINUS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_LINUS_URL))
+export MCGROF := $(subst ",,$(CONFIG_MIRROR_MCGROF_URL))
+export MCGROF_LINUS := $(subst ",,$(CONFIG_MIRROR_MCGROF_LINUS_URL))
+export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL))
+export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL))
+export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL))
+export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL))
+export KDEVOPS_FSTESTS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_FSTESTS_URL))
+export BLKTESTS := $(subst ",,$(CONFIG_MIRROR_BLKTESTS_URL))
+
LINUX_SERVICES := linux-mirror.service
LINUX_SERVICES += linux-stable-mirror.service
LINUX_SERVICES += linux-next-mirror.service
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 15/18] mirrors: add new mirrors.yaml and add xfsprogs
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (13 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 14/18] mirrors: move mirror editing into one file Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 16/18] mirrors: add xfsdump mirror and use it Luis Chamberlain
` (3 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
The process to add mirrors is pretty error prone. A lot of it
is because we're doing it the old way of shell scripts to scale
and we can do better. Just use kconfig to define preferences,
ansible to run tasks, and define a yaml file with the mirror
preferences outlined based on the target variability the user
has opted into.
We no longer have to touch the linux-mirror-systemd/Makefile
with clunky stuff, all we need to do to add new mirrors is edit
playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 and ensure
we have sensible defaults. This also lets us do variability.
The entire process to install user units and timers, and also
clone the mirrors is now done with python scripts that read the
input yaml file.
Also make use of this now in one shot. It will be clearer how
to add new mirrors in one shot in the next commit where we don't
add the infrastructure for it.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.gitignore | 5 +
Kconfig | 1 +
Makefile | 2 -
Makefile.linux-mirror | 26 ++-
kconfigs/Kconfig.defaults | 3 +
kconfigs/Kconfig.mirror | 4 +
playbooks/linux-mirror.yml | 4 +
playbooks/roles/linux-mirror/README.md | 9 ++
.../roles/linux-mirror/defaults/main.yml | 5 +
.../linux-mirror/python/gen-mirror-files.py | 149 ++++++++++++++++++
.../linux-mirror/python/start-mirroring.py | 110 +++++++++++++
playbooks/roles/linux-mirror/tasks/main.yml | 91 +++++++++++
.../linux-mirror/templates/mirrors.yaml.j2 | 4 +
workflows/fstests/xfs/Kconfig | 9 +-
14 files changed, 405 insertions(+), 17 deletions(-)
create mode 100644 kconfigs/Kconfig.defaults
create mode 100644 playbooks/linux-mirror.yml
create mode 100644 playbooks/roles/linux-mirror/README.md
create mode 100644 playbooks/roles/linux-mirror/defaults/main.yml
create mode 100755 playbooks/roles/linux-mirror/python/gen-mirror-files.py
create mode 100755 playbooks/roles/linux-mirror/python/start-mirroring.py
create mode 100644 playbooks/roles/linux-mirror/tasks/main.yml
create mode 100644 playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
diff --git a/.gitignore b/.gitignore
index ef5d5983ed13..af7d08a66d28 100644
--- a/.gitignore
+++ b/.gitignore
@@ -72,6 +72,11 @@ workflows/demos/reboot-limit/results/
workflows/gitr/results/
workflows/ltp/results/
+playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml
+# We should strive to ignore all and port them to the new mirrors.yaml
+playbooks/roles/linux-mirror/linux-mirror-systemd/xfsprogs-mirror.service
+playbooks/roles/linux-mirror/linux-mirror-systemd/xfsprogs-mirror.timer
+
# For now we ignore all results as we have work to do:
# - copy only data for failed tests
# - envision a way so that regressions are clearly visible with a git diff
diff --git a/Kconfig b/Kconfig
index 292f66e7e1a6..dab26fcc351a 100644
--- a/Kconfig
+++ b/Kconfig
@@ -23,6 +23,7 @@ config KDEVOPS_FIRST_RUN
for a series of entries which will help you with this first run.
Enabling these are not required, they do however help.
+source "kconfigs/Kconfig.defaults"
source "kconfigs/Kconfig.distro"
menu "Target architecture"
diff --git a/Makefile b/Makefile
index 5b8e1a22d872..cd4e639a4b01 100644
--- a/Makefile
+++ b/Makefile
@@ -148,9 +148,7 @@ ifeq (y,$(CONFIG_HYPERVISOR_TUNING))
include Makefile.hypervisor-tunings
endif # CONFIG_HYPERVISOR_TUNING
-ifeq (y,$(CONFIG_INSTALL_LOCAL_LINUX_MIRROR))
include Makefile.linux-mirror
-endif
ifeq (y,$(CONFIG_KDEVOPS_DISTRO_REG_METHOD_TWOLINE))
DEFAULT_DEPS += playbooks/secret.yml
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index c355c4a0ea98..d9515353f77b 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -1,24 +1,21 @@
# SPDX-License-Identifier: copyleft-next-0.3.1
+ifeq (y,$(CONFIG_INSTALL_LOCAL_LINUX_MIRROR))
LINUX_MIRROR_ARGS :=
LINUX_MIRROR_ARGS += install_linux_mirror=True
+LINUX_MIRROR_ARGS += defaults_xfsprogs_git='$(subst ",,$(CONFIG_DEFAULT_XFSPROGS_URL))'
MIRROR_CODE := $(TOPDIR)/playbooks/roles/linux-mirror/linux-mirror-systemd/
-# We don't use jinja2 yet to parse this extra_vars stuff to allow
-# customizations as the defaults seem sensible. However once and if
-# folks want to start adding variability to the how we deploy the
-# linux-mirror Ansible can be used:
-#kdevops_linux_mirror: $(KDEVOPS_EXTRA_VARS)
-# $(Q)ansible-playbook --connection=local \
-# --inventory localhost, \
-# $(KDEVOPS_PLAYBOOKS_DIR)/linux-mirror.yml \
-# -e 'ansible_python_interpreter=/usr/bin/python3' \
-# --extra-vars=@./extra_vars.yaml
-#
-
-# Since we don't support varabiliity yet, for now we keep it very simple:
-mirror: $(KDEVOPS_EXTRA_VARS)
+kdevops_linux_mirror: $(KDEVOPS_EXTRA_VARS)
+ $(Q)ansible-playbook --connection=local \
+ --inventory localhost, \
+ $(KDEVOPS_PLAYBOOKS_DIR)/linux-mirror.yml \
+ -e 'ansible_python_interpreter=/usr/bin/python3' \
+ --extra-vars=@./extra_vars.yaml
+PHONY += kdevops_linux_mirror
+
+mirror: $(KDEVOPS_EXTRA_VARS) kdevops_linux_mirror
$(Q)echo Setting up all mirrors on localhost
$(Q)make -C $(MIRROR_CODE) mirror
$(Q)make -C $(MIRROR_CODE) install
@@ -40,3 +37,4 @@ mirror-help-menu:
@echo ""
HELP_TARGETS += mirror-help-menu
+endif
diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
new file mode 100644
index 000000000000..ba5d6a3a78d6
--- /dev/null
+++ b/kconfigs/Kconfig.defaults
@@ -0,0 +1,3 @@
+config DEFAULT_XFSPROGS_URL
+ string
+ default "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index 7e17d59a605c..66f75cdd2c89 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -464,5 +464,9 @@ config MIRROR_BLKTESTS_URL
string
default "https://github.com/osandov/blktests.git"
+config MIRROR_XFSPROGS_URL
+ string
+ default DEFAULT_XFSPROGS_URL
+
endif # ENABLE_LOCAL_LINUX_MIRROR
endif # TERRAFORM
diff --git a/playbooks/linux-mirror.yml b/playbooks/linux-mirror.yml
new file mode 100644
index 000000000000..0808ebc1df80
--- /dev/null
+++ b/playbooks/linux-mirror.yml
@@ -0,0 +1,4 @@
+---
+- hosts: localhost
+ roles:
+ - role: linux-mirror
diff --git a/playbooks/roles/linux-mirror/README.md b/playbooks/roles/linux-mirror/README.md
new file mode 100644
index 000000000000..64df3566b48f
--- /dev/null
+++ b/playbooks/roles/linux-mirror/README.md
@@ -0,0 +1,9 @@
+linux_mirror
+============
+
+This lets us get us install git mirrors for our nodes.
+
+License
+-------
+
+copyleft-next
diff --git a/playbooks/roles/linux-mirror/defaults/main.yml b/playbooks/roles/linux-mirror/defaults/main.yml
new file mode 100644
index 000000000000..19ae271e4edc
--- /dev/null
+++ b/playbooks/roles/linux-mirror/defaults/main.yml
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier copyleft-next-0.3.1
+---
+mirror_user_system_dir: "~/.config/systemd/user/"
+install_linux_mirror: False
+defaults_xfsprogs_git: "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
diff --git a/playbooks/roles/linux-mirror/python/gen-mirror-files.py b/playbooks/roles/linux-mirror/python/gen-mirror-files.py
new file mode 100755
index 000000000000..d1096994e206
--- /dev/null
+++ b/playbooks/roles/linux-mirror/python/gen-mirror-files.py
@@ -0,0 +1,149 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# Generates mirror systemd service and timer files
+
+import argparse
+import yaml
+import json
+import sys
+import pprint
+import subprocess
+import time
+import os
+from pathlib import Path
+
+topdir = os.environ.get('TOPDIR', '.')
+yaml_dir = topdir + "/playbooks/roles/linux-mirror/linux-mirror-systemd/"
+default_mirrors_yaml = yaml_dir + 'mirrors.yaml'
+
+service_template = """[Unit]
+Description={short_name} mirror [{target}]
+Documentation=man:git(1)
+ConditionPathExists=/mirror/{target}
+
+[Service]
+Type=oneshot
+ExecStartPre=/usr/bin/git -C /mirror/{target} remote update --prune
+ExecStart=/usr/bin/git -C /mirror/{target} fetch --tags --prune
+ExecStartPost=/usr/bin/git -C /mirror/{target} fetch origin +refs/heads/*:refs/heads/*
+
+[Install]
+WantedBy=multi-user.target
+"""
+
+timer_template = """[Unit]
+Description={short_name} mirror query timer [{target}]
+ConditionPathExists=/mirror/{target}
+
+[Timer]
+OnBootSec={refresh_on_boot}
+OnUnitInactiveSec={refresh}
+
+[Install]
+WantedBy=default.target
+"""
+
+def main():
+ parser = argparse.ArgumentParser(description='gen-mirror-files')
+ parser.add_argument('--yaml-mirror', metavar='<yaml_mirror>', type=str,
+ default=default_mirrors_yaml,
+ help='The yaml mirror input file.')
+ parser.add_argument('--verbose', const=True, default=False, action="store_const",
+ help='Be verbose on otput.')
+ parser.add_argument('--refresh', metavar='<refresh>', type=str,
+ default='360m',
+ help='How often to update the git tree.')
+ parser.add_argument('--refresh-on-boot', metavar='<refresh>', type=str,
+ default='10m',
+ help='How long to wait on boot to update the git tree.')
+ args = parser.parse_args()
+
+ if not os.path.isfile(args.yaml_mirror):
+ sys.stdout.write("%s does not exist\n" % (yaml_mirror))
+ sys.exit(1)
+
+ # load the yaml input file
+ with open(f'{args.yaml_mirror}') as stream:
+ yaml_vars = yaml.safe_load(stream)
+
+ if yaml_vars.get('mirrors') is None:
+ raise Exception(f"Missing mirrors descriptions on %s" %
+ (args.yaml_mirror))
+
+ if (args.verbose):
+ sys.stdout.write("Yaml mirror input: %s\n\n" % args.yaml_mirror)
+
+ total = 0
+ for mirror in yaml_vars['mirrors']:
+ total = total + 1
+
+ if mirror.get('short_name') is None:
+ raise Exception(f"Missing required short_name on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+ if mirror.get('url') is None:
+ raise Exception(f"Missing required url on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+ if mirror.get('target') is None:
+ raise Exception(f"Missing required target on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+
+ short_name = mirror['short_name']
+ url = mirror['url']
+ target = mirror['target']
+
+ service_file = f"{yaml_dir}" + short_name + "-mirror" + ".service"
+ timer_file = f"{yaml_dir}" + short_name + "-mirror" + ".timer"
+
+ refresh = args.refresh
+ if mirror.get('refresh'):
+ refresh = mirror.get('refresh')
+ refresh_on_boot = args.refresh_on_boot
+ if mirror.get('refresh_on_boot'):
+ refresh = mirror.get('refresh_on_boot')
+
+ if (args.verbose):
+ sys.stdout.write("Mirror #%d\n" % total)
+ sys.stdout.write("\tshort_name: %s\n" % (mirror['short_name']))
+ sys.stdout.write("\turl: %s\n" % (mirror['short_name']))
+ sys.stdout.write("\ttarget: %s\n" % (mirror['short_name']))
+ sys.stdout.write("\tservice: %s\n" % (service_file))
+ sys.stdout.write("\ttimer: %s\n" % (timer_file))
+ sys.stdout.write("\trefresh: %s\n" % (refresh))
+ sys.stdout.write("\trefresh_on_boot: %s\n" % (refresh_on_boot))
+
+ if os.path.exists(service_file):
+ if (args.verbose):
+ sys.stdout.write("\toverwrite_service: True\n")
+ os.remove(service_file)
+ else:
+ if (args.verbose):
+ sys.stdout.write("\toverwrite_service: False\n")
+
+ output_service = open(service_file, 'w')
+ context = {
+ "short_name" : short_name,
+ "url" : url,
+ "target" : target,
+ }
+ output_service.write(service_template.format(**context))
+ output_service.close()
+
+ if os.path.exists(timer_file):
+ if (args.verbose):
+ sys.stdout.write("\toverwrite_timer: True\n")
+ os.remove(timer_file)
+ else:
+ if (args.verbose):
+ sys.stdout.write("\toverwrite_timer: False\n")
+
+ output_timer = open(timer_file, 'w')
+ context = {
+ "short_name" : short_name,
+ "url" : url,
+ "target" : target,
+ "refresh" : refresh,
+ "refresh_on_boot" : refresh_on_boot,
+ }
+ output_timer.write(timer_template.format(**context))
+ output_timer.close()
+
+if __name__ == "__main__":
+ main()
diff --git a/playbooks/roles/linux-mirror/python/start-mirroring.py b/playbooks/roles/linux-mirror/python/start-mirroring.py
new file mode 100755
index 000000000000..d7cb746f5bc0
--- /dev/null
+++ b/playbooks/roles/linux-mirror/python/start-mirroring.py
@@ -0,0 +1,110 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# Start mirroring based on mirrors.yaml file
+
+import argparse
+import yaml
+import json
+import sys
+import pprint
+import subprocess
+import time
+import os
+from pathlib import Path
+import subprocess
+
+topdir = os.environ.get('TOPDIR', '.')
+yaml_dir = topdir + "/playbooks/roles/linux-mirror/linux-mirror-systemd/"
+default_mirrors_yaml = yaml_dir + 'mirrors.yaml'
+
+mirror_path = '/mirror/'
+
+def main():
+ parser = argparse.ArgumentParser(description='start-mirroring')
+ parser.add_argument('--yaml-mirror', metavar='<yaml_mirror>', type=str,
+ default=default_mirrors_yaml,
+ help='The yaml mirror input file.')
+ parser.add_argument('--verbose', const=True, default=False, action="store_const",
+ help='Be verbose on otput.')
+ args = parser.parse_args()
+
+ if not os.path.isfile(args.yaml_mirror):
+ sys.stdout.write("%s does not exist\n" % (yaml_mirror))
+ sys.exit(1)
+
+ # load the yaml input file
+ with open(f'{args.yaml_mirror}') as stream:
+ yaml_vars = yaml.safe_load(stream)
+
+ if yaml_vars.get('mirrors') is None:
+ raise Exception(f"Missing mirrors descriptions on %s" %
+ (args.yaml_mirror))
+
+ if (args.verbose):
+ sys.stdout.write("Yaml mirror input: %s\n\n" % args.yaml_mirror)
+
+ total = 0
+ for mirror in yaml_vars['mirrors']:
+ total = total + 1
+
+ if mirror.get('short_name') is None:
+ raise Exception(f"Missing required short_name on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+ if mirror.get('url') is None:
+ raise Exception(f"Missing required url on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+ if mirror.get('target') is None:
+ raise Exception(f"Missing required target on mirror item #%d on file: %s" % (total, args.yaml_mirror))
+
+ short_name = mirror['short_name']
+ url = mirror['url']
+ target = mirror['target']
+ reference = None
+ reference_args = []
+
+ if mirror.get('reference'):
+ reference = mirror.get('reference')
+ reference_args = [ '--reference', reference ]
+
+ if (args.verbose):
+ sys.stdout.write("Mirror #%d\n" % total)
+ sys.stdout.write("\tshort_name: %s\n" % (short_name))
+ sys.stdout.write("\turl: %s\n" % (url))
+ sys.stdout.write("\ttarget: %s\n" % (url))
+ if reference is None:
+ sys.stdout.write("\treference: %s\n" % ("None"))
+ else:
+ sys.stdout.write("\treference: %s\n" % (reference))
+ cmd = [
+ 'git',
+ '-C',
+ mirror_path,
+ 'clone',
+ '--verbose',
+ '--progress',
+ '--mirror',
+ url,
+ target ]
+ cmd = cmd + reference_args
+ mirror_target = mirror_path + target
+ if os.path.isdir(mirror_target):
+ continue
+ sys.stdout.write("Mirroring: %s onto %s\n" % (short_name, mirror_target))
+ if (args.verbose):
+ sys.stdout.write("%s\n" % (cmd))
+ sys.stdout.write("%s\n" % (" ".join(cmd)))
+ process = subprocess.Popen(cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True)
+ try:
+ data = process.communicate(timeout=12000)
+ except subprocess.TimeoutExpired:
+ return "Timeout"
+ else:
+ process.wait()
+ if process.returncode != 0:
+ raise Exception(f"Failed clone with:\n%s" % (" ".join(cmd)))
+
+if __name__ == "__main__":
+ main()
diff --git a/playbooks/roles/linux-mirror/tasks/main.yml b/playbooks/roles/linux-mirror/tasks/main.yml
new file mode 100644
index 000000000000..461e131c93fd
--- /dev/null
+++ b/playbooks/roles/linux-mirror/tasks/main.yml
@@ -0,0 +1,91 @@
+---
+- name: Import optional extra_args file
+ include_vars: "{{ item }}"
+ ignore_errors: yes
+ with_first_found:
+ - files:
+ - "../extra_vars.yml"
+ - "../extra_vars.yaml"
+ - "../extra_vars.json"
+ skip: true
+ tags: vars
+
+- name: Set up the mirrors.yaml based on preferences configured
+ tags: [ 'mirror' ]
+ template:
+ src: "mirrors.yaml.j2"
+ dest: "{{ topdir_path }}/playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml"
+ force: yes
+ trim_blocks: True
+ lstrip_blocks: True
+ when:
+ - install_linux_mirror|bool
+
+- name: Start mirroring
+ delegate_to: localhost
+ run_once: true
+ shell: |
+ set -o pipefail
+ {{ role_path }}/python/start-mirroring.py --verbose
+ args:
+ executable: /bin/bash
+ chdir: "{{ topdir_path }}"
+ when:
+ - install_linux_mirror|bool
+
+- name: Generate systemd service and timer unit files
+ delegate_to: localhost
+ run_once: true
+ shell: |
+ set -o pipefail
+ {{ role_path }}/python/gen-mirror-files.py --refresh 360m --refresh-on-boot 10m
+ args:
+ executable: /bin/bash
+ chdir: "{{ topdir_path }}"
+ when:
+ - install_linux_mirror|bool
+
+- name: load variables from yaml file
+ include_vars:
+ file: "{{ topdir_path }}/playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml"
+ name: mirrors
+
+- name: Copy systemd service file to user systemd dir
+ copy:
+ src: "{{ topdir_path }}/playbooks/roles/linux-mirror/linux-mirror-systemd/{{ item.short_name}}-mirror.service"
+ dest: "{{ mirror_user_system_dir }}/"
+ with_items:
+ - "{{ mirrors.get('mirrors') }}"
+ loop_control:
+ label: "Installing {{ item.short_name}}-mirror.service user serice unit ..."
+
+- name: Copy systemd timer file to user systemd dir
+ copy:
+ src: "{{ topdir_path }}/playbooks/roles/linux-mirror/linux-mirror-systemd/{{ item.short_name}}-mirror.timer"
+ dest: "{{ mirror_user_system_dir }}/"
+ with_items:
+ - "{{ mirrors.get('mirrors') }}"
+ loop_control:
+ label: "Installing {{ item.short_name}}-mirror.service user unit ..."
+
+- name: Enable systemd unit for mirror services
+ ansible.builtin.systemd_service:
+ name: "{{ item.short_name}}-mirror.service"
+ state: started
+ enabled: true
+ scope: user
+ with_items:
+ - "{{ mirrors.get('mirrors') }}"
+ loop_control:
+ label: "Enabling {{ item.short_name}}-mirror.service unit ..."
+
+- name: Enable systemd unit for mirror timers
+ ansible.builtin.systemd_service:
+ name: "{{ item.short_name }}-mirror.timer"
+ state: started
+ enabled: true
+ scope: user
+ with_items:
+ - "{{ mirrors.get('mirrors') }}"
+ loop_control:
+ label: "Enabling {{ item.short_name}}-mirror.timer unit ..."
diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
new file mode 100644
index 000000000000..7151873783d0
--- /dev/null
+++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
@@ -0,0 +1,4 @@
+mirrors:
+ - short_name: "xfsprogs"
+ url: "{{ defaults_xfsprogs_git }}"
+ target: "xfsprogs-dev.git"
diff --git a/workflows/fstests/xfs/Kconfig b/workflows/fstests/xfs/Kconfig
index 28fe44eb98f9..b638d1097656 100644
--- a/workflows/fstests/xfs/Kconfig
+++ b/workflows/fstests/xfs/Kconfig
@@ -35,9 +35,16 @@ config FSTESTS_XFS_BUILD_CUSTOM_XFSPROGS
help
Enable this option to build and install a custom version of Xfsprogs.
+config HAVE_MIRROR_XFSPROGS
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/xfsprogs-dev.git)
+
config FSTESTS_XFS_XFSPROGS_GIT
string "The Xfsprogs git tree to clone"
- default "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
+ default DEFAULT_XFSPROGS_URL if !HAVE_MIRROR_XFSPROGS
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/xfsprogs-dev.git) if HAVE_MIRROR_XFSPROGS && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/xfsprogs-dev.git) if HAVE_MIRROR_XFSPROGS && GUESTFS
depends on FSTESTS_XFS_BUILD_CUSTOM_XFSPROGS
help
The Xfsprogs git tree to clone
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 16/18] mirrors: add xfsdump mirror and use it
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (14 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 15/18] mirrors: add new mirrors.yaml and add xfsprogs Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 17/18] mirrors: add dbench git Luis Chamberlain
` (2 subsequent siblings)
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Now that we have an easy way to support mirrors, let's just
add xfsdump in one shot and enable it too.
Note: only debian uses this git tree.
We check that we get the right URL when XFS is enabled and the
mirror has been cloned:
grep dump extra_vars.yaml
fstests_xsfsdump_git: git://192.168.122.1/mirror/xfsdump-dev.git
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile.linux-mirror | 1 +
kconfigs/Kconfig.defaults | 7 +++++++
playbooks/roles/fstests/defaults/main.yml | 2 ++
.../roles/fstests/tasks/install-deps/debian/main.yml | 2 +-
playbooks/roles/linux-mirror/defaults/main.yml | 1 +
.../roles/linux-mirror/templates/mirrors.yaml.j2 | 3 +++
workflows/fstests/Makefile.xfs | 2 ++
workflows/fstests/xfs/Kconfig | 11 +++++++++++
8 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror
index d9515353f77b..67bb463da53e 100644
--- a/Makefile.linux-mirror
+++ b/Makefile.linux-mirror
@@ -4,6 +4,7 @@ ifeq (y,$(CONFIG_INSTALL_LOCAL_LINUX_MIRROR))
LINUX_MIRROR_ARGS :=
LINUX_MIRROR_ARGS += install_linux_mirror=True
LINUX_MIRROR_ARGS += defaults_xfsprogs_git='$(subst ",,$(CONFIG_DEFAULT_XFSPROGS_URL))'
+LINUX_MIRROR_ARGS += defaults_xfsdump_git='$(subst ",,$(CONFIG_DEFAULT_XFSDUMP_URL))'
MIRROR_CODE := $(TOPDIR)/playbooks/roles/linux-mirror/linux-mirror-systemd/
diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
index ba5d6a3a78d6..e559f28f964d 100644
--- a/kconfigs/Kconfig.defaults
+++ b/kconfigs/Kconfig.defaults
@@ -1,3 +1,10 @@
+# Please keep in alphabetical order
+
+config DEFAULT_XFSDUMP_URL
+ string
+ default "https://github.com/linux-kdevops/xfsdump-dev.git"
+
config DEFAULT_XFSPROGS_URL
string
default "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
+
diff --git a/playbooks/roles/fstests/defaults/main.yml b/playbooks/roles/fstests/defaults/main.yml
index a46a4672dfa5..c3df0da2718a 100644
--- a/playbooks/roles/fstests/defaults/main.yml
+++ b/playbooks/roles/fstests/defaults/main.yml
@@ -184,3 +184,5 @@ fstests_tmpfs_section_huge_within_size: False
fstests_tmpfs_section_huge_advise: False
devconfig_enable_systemd_journal_remote: False
+
+fstests_xsfsdump_git: "https://github.com/linux-kdevops/xfsdump-dev.git"
diff --git a/playbooks/roles/fstests/tasks/install-deps/debian/main.yml b/playbooks/roles/fstests/tasks/install-deps/debian/main.yml
index 1b00b88cb5fd..d806cd678818 100644
--- a/playbooks/roles/fstests/tasks/install-deps/debian/main.yml
+++ b/playbooks/roles/fstests/tasks/install-deps/debian/main.yml
@@ -94,7 +94,7 @@
# in Debian Testing repository
- name: git clone xfsdump
git:
- repo: "https://github.com/linux-kdevops/xfsdump-dev.git"
+ repo: "{{ fstests_xsfsdump_git }}"
dest: "{{ data_path }}/xfsdump"
update: yes
version: v3.1.10-fixed
diff --git a/playbooks/roles/linux-mirror/defaults/main.yml b/playbooks/roles/linux-mirror/defaults/main.yml
index 19ae271e4edc..461f47cfedf9 100644
--- a/playbooks/roles/linux-mirror/defaults/main.yml
+++ b/playbooks/roles/linux-mirror/defaults/main.yml
@@ -3,3 +3,4 @@
mirror_user_system_dir: "~/.config/systemd/user/"
install_linux_mirror: False
defaults_xfsprogs_git: "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
+defaults_xfsdump_git: "https://github.com/linux-kdevops/xfsdump-dev.git"
diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
index 7151873783d0..2f0267c27c7b 100644
--- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
+++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
@@ -2,3 +2,6 @@ mirrors:
- short_name: "xfsprogs"
url: "{{ defaults_xfsprogs_git }}"
target: "xfsprogs-dev.git"
+ - short_name: "xfsdump"
+ url: "{{ defaults_xfsdump_git }}"
+ target: "xfsdump-dev.git"
diff --git a/workflows/fstests/Makefile.xfs b/workflows/fstests/Makefile.xfs
index 54194f243876..542946e12f9c 100644
--- a/workflows/fstests/Makefile.xfs
+++ b/workflows/fstests/Makefile.xfs
@@ -6,6 +6,8 @@ FSTESTS_ARGS += fstests_xfs_xfsprogs_data='$(CONFIG_FSTESTS_XFS_XFSPROGS_DATA)'
FSTESTS_ARGS += fstests_xfs_xfsprogs_install_path='$(CONFIG_FSTESTS_XFS_XFSPROGS_INSTALL_PATH)'
endif
+FSTESTS_ARGS += fstests_xsfsdump_git='$(subst ",,$(CONFIG_FSTESTS_XFSDUMP_GIT_URL))'
+
ifeq (y,$(CONFIG_FSTESTS_XFS_QUOTA_ENABLED))
FSTESTS_ARGS += fstests_xfs_mount_quota_opts='$(CONFIG_FSTESTS_XFS_MOUNT_QUOTA_OPTS)'
endif
diff --git a/workflows/fstests/xfs/Kconfig b/workflows/fstests/xfs/Kconfig
index b638d1097656..9bc787f28896 100644
--- a/workflows/fstests/xfs/Kconfig
+++ b/workflows/fstests/xfs/Kconfig
@@ -68,6 +68,17 @@ config FSTESTS_XFS_XFSPROGS_INSTALL_PATH
help
Destination path to install Xfsprogs binaries
+config HAVE_MIRROR_XFSDUMP
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/xfsdump-dev.git)
+
+config FSTESTS_XFSDUMP_GIT_URL
+ string
+ default DEFAULT_XFSDUMP_URL if !HAVE_MIRROR_XFSDUMP
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/xfsdump-dev.git) if HAVE_MIRROR_XFSDUMP && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/xfsdump-dev.git) if HAVE_MIRROR_XFSDUMP && GUESTFS
+
config FSTESTS_XFS_QUOTA_ENABLED
bool "Enable XFS quota"
default n
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 17/18] mirrors: add dbench git
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (15 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 16/18] mirrors: add xfsdump mirror and use it Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 2:25 ` [PATCH 18/18] mirrors: add blktrace mirror and use it Luis Chamberlain
2024-03-30 16:37 ` [PATCH 00/18] mirrors: enhance and add tons of mirrors Chuck Lever III
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
We're actually the main source of the latest dbench so just use that.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
kconfigs/Kconfig.defaults | 4 ++++
playbooks/roles/blktests/defaults/main.yml | 1 +
playbooks/roles/blktests/tasks/main.yml | 2 +-
playbooks/roles/linux-mirror/defaults/main.yml | 1 +
.../roles/linux-mirror/templates/mirrors.yaml.j2 | 3 +++
workflows/blktests/Kconfig | 11 +++++++++++
workflows/blktests/Makefile | 1 +
7 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
index e559f28f964d..d552735bdb3b 100644
--- a/kconfigs/Kconfig.defaults
+++ b/kconfigs/Kconfig.defaults
@@ -1,5 +1,9 @@
# Please keep in alphabetical order
+config DEFAULT_DBENCH_URL
+ string
+ default "https://github.com/linux-kdevops/dbench.git"
+
config DEFAULT_XFSDUMP_URL
string
default "https://github.com/linux-kdevops/xfsdump-dev.git"
diff --git a/playbooks/roles/blktests/defaults/main.yml b/playbooks/roles/blktests/defaults/main.yml
index dd009f58b3a3..8c50bf6d8e63 100644
--- a/playbooks/roles/blktests/defaults/main.yml
+++ b/playbooks/roles/blktests/defaults/main.yml
@@ -22,3 +22,4 @@ nbd_version: "nbd-3.21"
nbd_data: "{{data_path}}/nbd"
compile_dbench: False
+blktests_dbench_git: "https://github.com/linux-kdevops/dbench.git"
diff --git a/playbooks/roles/blktests/tasks/main.yml b/playbooks/roles/blktests/tasks/main.yml
index fd4943b36088..590f981af0b8 100644
--- a/playbooks/roles/blktests/tasks/main.yml
+++ b/playbooks/roles/blktests/tasks/main.yml
@@ -42,7 +42,7 @@
name: compile_dbench
vars:
dbench_data: "{{ data_path }}/dbench"
- dbench_git: "https://github.com/linux-kdevops/dbench.git"
+ dbench_git: "{{ blktests_dbench_git }}"
tags: [ 'oscheck', 'git', 'dbench']
- name: Remove any old blktrace as we always clone a fresh tree
diff --git a/playbooks/roles/linux-mirror/defaults/main.yml b/playbooks/roles/linux-mirror/defaults/main.yml
index 461f47cfedf9..e0788bb43c94 100644
--- a/playbooks/roles/linux-mirror/defaults/main.yml
+++ b/playbooks/roles/linux-mirror/defaults/main.yml
@@ -4,3 +4,4 @@ mirror_user_system_dir: "~/.config/systemd/user/"
install_linux_mirror: False
defaults_xfsprogs_git: "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
defaults_xfsdump_git: "https://github.com/linux-kdevops/xfsdump-dev.git"
+defaults_dbench_git: "https://github.com/linux-kdevops/dbench.git"
diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
index 2f0267c27c7b..0389f3d76396 100644
--- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
+++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
@@ -5,3 +5,6 @@ mirrors:
- short_name: "xfsdump"
url: "{{ defaults_xfsdump_git }}"
target: "xfsdump-dev.git"
+ - short_name: "dbench"
+ url: "{{ defaults_dbench_git }}"
+ target: "dbench.git"
diff --git a/workflows/blktests/Kconfig b/workflows/blktests/Kconfig
index 2be13de90a76..05e16bcf79cd 100644
--- a/workflows/blktests/Kconfig
+++ b/workflows/blktests/Kconfig
@@ -12,6 +12,17 @@ config HAVE_DISTRO_PREFERS_BLKTESTS_WATCHDOG_RESET
bool
default n
+config HAVE_MIRROR_DBENCH
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/dbench.git)
+
+config BLKTESTS_DBENCH_GIT_URL
+ string
+ default DEFAULT_DBENCH_URL if !HAVE_MIRROR_DBENCH
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/dbench.git) if HAVE_MIRROR_XFSDUMP && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/dbench.git) if HAVE_MIRROR_XFSDUMP && GUESTFS
+
config BLKTESTS_WATCHDOG
bool "Enable kdevops blktests watchdog"
default y if HAVE_DISTRO_PREFERS_BLKTESTS_WATCHDOG
diff --git a/workflows/blktests/Makefile b/workflows/blktests/Makefile
index 829694af8bd3..ac4e8c2d1831 100644
--- a/workflows/blktests/Makefile
+++ b/workflows/blktests/Makefile
@@ -27,6 +27,7 @@ NBD_DATA:=$(subst ",,$(CONFIG_NBD_DATA))
BLKTESTS_DATA_TARGET:=$(subst ",,$(CONFIG_BLKTESTS_DATA_TARGET))
+BLKTESTS_ARGS += blktests_dbench_git='$(subst ",,$(CONFIG_BLKTESTS_DBENCH_GIT_URL))'
BLKTESTS_ARGS += blktests_git='$(BLKTESTS_GIT)'
BLKTESTS_ARGS += blktests_data=\"$(BLKTESTS_DATA)\"
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 18/18] mirrors: add blktrace mirror and use it
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (16 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 17/18] mirrors: add dbench git Luis Chamberlain
@ 2024-03-30 2:25 ` Luis Chamberlain
2024-03-30 16:37 ` [PATCH 00/18] mirrors: enhance and add tons of mirrors Chuck Lever III
18 siblings, 0 replies; 22+ messages in thread
From: Luis Chamberlain @ 2024-03-30 2:25 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Now that adding mirrors is rather easier than before add the
mirror for blktrace.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
kconfigs/Kconfig.defaults | 4 ++++
playbooks/roles/linux-mirror/defaults/main.yml | 1 +
playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 | 3 +++
workflows/blktests/Kconfig | 9 ++++++++-
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
index d552735bdb3b..b52ec86127d0 100644
--- a/kconfigs/Kconfig.defaults
+++ b/kconfigs/Kconfig.defaults
@@ -1,5 +1,9 @@
# Please keep in alphabetical order
+config DEFAULT_BLKTRACE_URL
+ string
+ default "https://git.kernel.dk/blktrace"
+
config DEFAULT_DBENCH_URL
string
default "https://github.com/linux-kdevops/dbench.git"
diff --git a/playbooks/roles/linux-mirror/defaults/main.yml b/playbooks/roles/linux-mirror/defaults/main.yml
index e0788bb43c94..7c0d7bf623f2 100644
--- a/playbooks/roles/linux-mirror/defaults/main.yml
+++ b/playbooks/roles/linux-mirror/defaults/main.yml
@@ -5,3 +5,4 @@ install_linux_mirror: False
defaults_xfsprogs_git: "https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git"
defaults_xfsdump_git: "https://github.com/linux-kdevops/xfsdump-dev.git"
defaults_dbench_git: "https://github.com/linux-kdevops/dbench.git"
+defaults_blktrace_git: "https://git.kernel.dk/blktrace"
diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
index 0389f3d76396..c85700738a14 100644
--- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
+++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
@@ -8,3 +8,6 @@ mirrors:
- short_name: "dbench"
url: "{{ defaults_dbench_git }}"
target: "dbench.git"
+ - short_name: "blktrace"
+ url: "{{ defaults_blktrace_git }}"
+ target: "blktrace.git"
diff --git a/workflows/blktests/Kconfig b/workflows/blktests/Kconfig
index 05e16bcf79cd..998939d137f9 100644
--- a/workflows/blktests/Kconfig
+++ b/workflows/blktests/Kconfig
@@ -159,9 +159,16 @@ config BLKTESTS_DATA
Note that {{data_path}} corresponds to the location set by the
configuration option CONFIG_WORKFLOW_DATA_PATH.
+config HAVE_MIRROR_BLKTRACE
+ bool
+ depends on USE_LIBVIRT_MIRROR
+ default $(shell, scripts/check_mirror_present.sh /mirror/blktrace.git)
+
config BLKTRACE_GIT
string "The blktrace git tree to clone"
- default "https://git.kernel.dk/blktrace"
+ default DEFAULT_BLKTRACE_URL if !HAVE_MIRROR_BLKTRACE
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_VAGRANT) /mirror/blktrace.git) if HAVE_MIRROR_BLKTRACE && VAGRANT
+ default $(shell, scripts/append-makefile-vars.sh git:// $(KDEVOPS_DEFAULT_BRIDGE_IP_GUESTFS) /mirror/blktrace.git) if HAVE_MIRROR_BLKTRACE && GUESTFS
help
The blktrace git tree to clone.
--
2.43.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH 00/18] mirrors: enhance and add tons of mirrors
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
` (17 preceding siblings ...)
2024-03-30 2:25 ` [PATCH 18/18] mirrors: add blktrace mirror and use it Luis Chamberlain
@ 2024-03-30 16:37 ` Chuck Lever III
2024-04-03 1:01 ` Luis Chamberlain
18 siblings, 1 reply; 22+ messages in thread
From: Chuck Lever III @ 2024-03-30 16:37 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: kdevops
> On Mar 29, 2024, at 10:25 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> If you are stuck in a world where you don't have much data for your
> downloads you have only one option: mirror everything. So the motivation
> for this is the exand the existing mirror support on kdevops to mirror
> more things other than just the kernel and qemu. We now can mirror
> *everything* we clone including kdevops itself.
>
> We also enhance this all so that we can *use* the mirrors by *default*
> if you have them and you are using local virtualization. We differentiate
> between vagrant and guestfs.
>
> Towards the end this just simplifies the process by adding a yaml file
> for the mirrors so that we can ingest them and process that with a few
> python scripts. The last commits demonstrate how easy it is now to add
> new mirrors.
>
> There's only few mirrors left to add but we can do that next week.
I've just begun to browse these. Seems like a good plan to be a
better netizen.
This comment falls into the category of beard-scratching, but I
wonder if there's a packaged git mirror we can use instead of
building our own? Just a random thought. Obviously some of these
changes are good to have anyway (like the bridge IP improvements).
--
Chuck Lever
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/18] mirrors: enhance and add tons of mirrors
2024-03-30 16:37 ` [PATCH 00/18] mirrors: enhance and add tons of mirrors Chuck Lever III
@ 2024-04-03 1:01 ` Luis Chamberlain
2024-04-03 18:04 ` Chuck Lever III
0 siblings, 1 reply; 22+ messages in thread
From: Luis Chamberlain @ 2024-04-03 1:01 UTC (permalink / raw)
To: Chuck Lever III; +Cc: kdevops
On Sat, Mar 30, 2024 at 04:37:47PM +0000, Chuck Lever III wrote:
>
>
> > On Mar 29, 2024, at 10:25 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > If you are stuck in a world where you don't have much data for your
> > downloads you have only one option: mirror everything. So the motivation
> > for this is the exand the existing mirror support on kdevops to mirror
> > more things other than just the kernel and qemu. We now can mirror
> > *everything* we clone including kdevops itself.
> >
> > We also enhance this all so that we can *use* the mirrors by *default*
> > if you have them and you are using local virtualization. We differentiate
> > between vagrant and guestfs.
> >
> > Towards the end this just simplifies the process by adding a yaml file
> > for the mirrors so that we can ingest them and process that with a few
> > python scripts. The last commits demonstrate how easy it is now to add
> > new mirrors.
> >
> > There's only few mirrors left to add but we can do that next week.
>
> I've just begun to browse these. Seems like a good plan to be a
> better netizen.
OK merged :)
> This comment falls into the category of beard-scratching, but I
> wonder if there's a packaged git mirror we can use instead of
> building our own? Just a random thought. Obviously some of these
> changes are good to have anyway (like the bridge IP improvements).
We'd have to agree on some random distro package for it. The
implemention for mirroring wasn't complex, a couple of python scripts
to read a yaml file, and which generate systemd unit service and timer
files. The big integration is with our Kconfig logic and so I think
its as slim as can be really. But if you find something we can opt-in
it as an alternative via Kconfig too.
Luis
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/18] mirrors: enhance and add tons of mirrors
2024-04-03 1:01 ` Luis Chamberlain
@ 2024-04-03 18:04 ` Chuck Lever III
0 siblings, 0 replies; 22+ messages in thread
From: Chuck Lever III @ 2024-04-03 18:04 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: kdevops
> On Apr 2, 2024, at 9:01 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Sat, Mar 30, 2024 at 04:37:47PM +0000, Chuck Lever III wrote:
>>
>>
>>> On Mar 29, 2024, at 10:25 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>>>
>>> If you are stuck in a world where you don't have much data for your
>>> downloads you have only one option: mirror everything. So the motivation
>>> for this is the exand the existing mirror support on kdevops to mirror
>>> more things other than just the kernel and qemu. We now can mirror
>>> *everything* we clone including kdevops itself.
>>>
>>> We also enhance this all so that we can *use* the mirrors by *default*
>>> if you have them and you are using local virtualization. We differentiate
>>> between vagrant and guestfs.
>>>
>>> Towards the end this just simplifies the process by adding a yaml file
>>> for the mirrors so that we can ingest them and process that with a few
>>> python scripts. The last commits demonstrate how easy it is now to add
>>> new mirrors.
>>>
>>> There's only few mirrors left to add but we can do that next week.
>>
>> I've just begun to browse these. Seems like a good plan to be a
>> better netizen.
>
> OK merged :)
>
>> This comment falls into the category of beard-scratching, but I
>> wonder if there's a packaged git mirror we can use instead of
>> building our own? Just a random thought. Obviously some of these
>> changes are good to have anyway (like the bridge IP improvements).
>
> We'd have to agree on some random distro package for it. The
> implemention for mirroring wasn't complex, a couple of python scripts
> to read a yaml file, and which generate systemd unit service and timer
> files. The big integration is with our Kconfig logic and so I think
> its as slim as can be really. But if you find something we can opt-in
> it as an alternative via Kconfig too.
I don't have anything specific in mind other than not
re-inventing the wheel if that would make our lives
harder. But sounds like it's a small thing and not
worth pulling in a packaged mirroring system.
--
Chuck Lever
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2024-04-03 18:04 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-30 2:25 [PATCH 00/18] mirrors: enhance and add tons of mirrors Luis Chamberlain
2024-03-30 2:25 ` [PATCH 01/18] linux-mirror: add kdevops mirroring support Luis Chamberlain
2024-03-30 2:25 ` [PATCH 02/18] Makefile.min_deps: add netcat as a min binary dependency Luis Chamberlain
2024-03-30 2:25 ` [PATCH 03/18] provision: generalize bridge-ip further Luis Chamberlain
2024-03-30 2:25 ` [PATCH 04/18] mirrors: add guestfs mirror options without 9p Luis Chamberlain
2024-03-30 2:25 ` [PATCH 05/18] mirrors: enable kdevops mirror Luis Chamberlain
2024-03-30 2:25 ` [PATCH 06/18] mirrors: add fstests to mirrors Luis Chamberlain
2024-03-30 2:25 ` [PATCH 07/18] fstests: use local primary fstests mirror if present Luis Chamberlain
2024-03-30 2:25 ` [PATCH 08/18] mirrors: add kdevops git alternative for fstests Luis Chamberlain
2024-03-30 2:25 ` [PATCH 09/18] mirrors: use kdevops fstests mirror when available Luis Chamberlain
2024-03-30 2:25 ` [PATCH 10/18] mirrors: add blktests mirror Luis Chamberlain
2024-03-30 2:25 ` [PATCH 11/18] mirrors: use blktests mirror when its available Luis Chamberlain
2024-03-30 2:25 ` [PATCH 12/18] mirrors: increase scope of mirroring in output Luis Chamberlain
2024-03-30 2:25 ` [PATCH 13/18] mirrors: move status check Luis Chamberlain
2024-03-30 2:25 ` [PATCH 14/18] mirrors: move mirror editing into one file Luis Chamberlain
2024-03-30 2:25 ` [PATCH 15/18] mirrors: add new mirrors.yaml and add xfsprogs Luis Chamberlain
2024-03-30 2:25 ` [PATCH 16/18] mirrors: add xfsdump mirror and use it Luis Chamberlain
2024-03-30 2:25 ` [PATCH 17/18] mirrors: add dbench git Luis Chamberlain
2024-03-30 2:25 ` [PATCH 18/18] mirrors: add blktrace mirror and use it Luis Chamberlain
2024-03-30 16:37 ` [PATCH 00/18] mirrors: enhance and add tons of mirrors Chuck Lever III
2024-04-03 1:01 ` Luis Chamberlain
2024-04-03 18:04 ` Chuck Lever III
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.