* [PATCH v1 0/3] fio: option to make engines load dynamically
@ 2020-06-28 19:27 Yigal Korman
2020-06-28 19:27 ` [PATCH v1 1/3] configure/Makefile: engine LIBS consistency Yigal Korman
` (3 more replies)
0 siblings, 4 replies; 16+ messages in thread
From: Yigal Korman @ 2020-06-28 19:27 UTC (permalink / raw)
To: axboe; +Cc: Yigal Korman, fio
Hi,
The following patchset provides a configuration option that converts a
select set of engines into external engines that are loaded dynamically
on demand. The purpose of this option is to provide distribution
packagers the ability to separate the core fio functionality from the
addition optional engines. Some of the optional engines have multiple
dependencies that increase the footprint of fio considerably even when
the user requires only the basic 'sync' engine. See examples of the
issue below.
I've tried to keep the changes as small and unobtrusive as possible.
The only significant change is the way an engine is defined in the
Makefile (variables are now prefixed with engine name). If the option
is provided, the Makefile will build the engines as external libraries
and install them in /usr/lib/fio by default. Fio will then search for
the engines in that location if the engine is not compiled in. The last
patch should help with the migration to the new scheme by suggesting
that an additional package is required if the engine is not found.
Future work to consider:
* Move the engine help text from fio to the engines themselves. Create
* a whitelist of engines so that only engines that are part of the main
* codebase are searched and loaded automatically. Convert more engines
* to this scheme. The current set of engines was selected based on
* simplicity and external dependencies. Engines with multiple exported
* flavors (i.e. glusterfs) are not converted yet.
Branch is also available on github, at
https://github.com/ykorman/fio/tree/dynamic-libengines.
A sample of how to package fio with this new mode for RPM-based
distributions is available here:
https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines
A sample of the packages for Fedora can be found here:
https://copr.fedorainfracloud.org/coprs/ykorman/fio/
See examples of how this improves our quality of life (tm) below the
pull summary.
Yigal Korman (3):
configure/Makefile: engine LIBS consistency
configure: new --dynamic-libengines build option
fio: suggest installing engine package when missing
.gitignore | 1 +
Makefile | 86 +++++++++++++++++++++++++++++++++++++---------
configure | 20 +++++------
engines/dev-dax.c | 2 +-
engines/guasi.c | 2 +-
engines/http.c | 2 +-
engines/libaio.c | 2 +-
engines/libhdfs.c | 6 ++--
engines/libiscsi.c | 6 ++--
engines/libpmem.c | 2 +-
engines/libzbc.c | 2 +-
engines/nbd.c | 2 +-
engines/pmemblk.c | 2 +-
engines/rados.c | 2 +-
engines/rbd.c | 2 +-
engines/rdma.c | 9 ++---
ioengines.c | 26 ++++++++++++--
ioengines.h | 6 ++++
os/os-linux.h | 2 ++
19 files changed, 131 insertions(+), 51 deletions(-)
--
2.17.1
Fedora 31 install of fio, before the the patches (redacted for brevity):
[root@host ~]# dnf install fio
Dependencies resolved.
==========================================================
Package Version Size
==========================================================
Installing:
fio 3.14-3.fc31 524 k
Installing dependencies:
cpio 2.12-12.fc31 260 k
daxctl-libs 68-1.fc31 38 k
dracut 050-26.git20200316.fc31 321 k
findutils 1:4.6.0-25.fc31 517 k
hwdata 0.336-1.fc31 1.4 M
kbd-legacy 2.0.4-14.fc31 464 k
kbd-misc 2.0.4-14.fc31 1.5 M
kmod 26-4.fc31 122 k
libaio 0.3.111-6.fc31 24 k
libibverbs 20.1-6.fc31 256 k
libkcapi 1.1.5-1.fc31 47 k
libkcapi-hmaccalc 1.1.5-1.fc31 29 k
libnl3 3.5.0-1.fc31 331 k
libpmem 1.7-1.fc31 80 k
libpmemblk 1.7-1.fc31 92 k
librados2 2:14.2.9-2.fc31 3.5 M
librbd1 2:14.2.9-2.fc31 1.8 M
librdmacm 20.1-6.fc31 66 k
lttng-ust 2.10.4-2.fc31 271 k
ndctl-libs 68-1.fc31 77 k
nspr 4.25.0-1.fc31 137 k
nss 3.53.0-1.fc31 677 k
nss-softokn 3.53.0-1.fc31 441 k
nss-softokn-freebl 3.53.0-1.fc31 248 k
nss-sysinit 3.53.0-1.fc31 20 k
nss-util 3.53.0-1.fc31 89 k
numactl-libs 2.0.12-3.fc31 30 k
pciutils 3.6.4-1.fc31 90 k
pciutils-libs 3.6.4-1.fc31 41 k
procps-ng 3.3.15-6.fc31 326 k
rdma-core 20.1-6.fc31 51 k
systemd-udev 243.8-1.fc31 1.3 M
userspace-rcu 0.11.1-2.fc31 104 k
xz 5.2.4-6.fc31 147 k
Installing weak dependenc
kbd 2.0.4-14.fc31 391 k
kpartx 0.8.0-3.fc31 55 k
pigz 2.4-5.fc31 78 k
systemd-bootchart 233-5.fc31 71 k
Transaction Summary
=========================================================
Install 39 Packages
Total download size: 16 M
Installed size: 57 M
After this patchset:
[root@host ~]# dnf install fio
===============================================
Package Version Size
===============================================
Installing:
fio 3.20-1.fc33 526 k
Installing dependencies:
numactl-libs 2.0.12-3.fc31 30 k
Transaction Summary
===============================================
Install 2 Packages
Total size: 557 k
Total download size: 30 k
Installed size: 2.1 M
And when a specific engine is needed:
[root@host ~]# dnf install fio-engine-rados
Dependencies resolved.
========================================================
Package Version Size
========================================================
Installing:
fio-engine-rados 3.20-1.fc33 15 k
Installing dependencies:
cpio 2.12-12.fc31 260 k
dracut 050-26.git20200316.fc31 321 k
findutils 1:4.6.0-25.fc31 517 k
hwdata 0.336-1.fc31 1.4 M
kbd-legacy 2.0.4-14.fc31 464 k
kbd-misc 2.0.4-14.fc31 1.5 M
kmod 26-4.fc31 122 k
libibverbs 20.1-6.fc31 256 k
libkcapi 1.1.5-1.fc31 47 k
libkcapi-hmaccalc 1.1.5-1.fc31 29 k
libnl3 3.5.0-1.fc31 331 k
librados2 2:14.2.9-2.fc31 3.5 M
librdmacm 20.1-6.fc31 66 k
lttng-ust 2.10.4-2.fc31 271 k
nspr 4.25.0-1.fc31 137 k
nss 3.53.0-1.fc31 677 k
nss-softokn 3.53.0-1.fc31 441 k
nss-softokn-freebl 3.53.0-1.fc31 248 k
nss-sysinit 3.53.0-1.fc31 20 k
nss-util 3.53.0-1.fc31 89 k
pciutils 3.6.4-1.fc31 90 k
pciutils-libs 3.6.4-1.fc31 41 k
procps-ng 3.3.15-6.fc31 326 k
rdma-core 20.1-6.fc31 51 k
systemd-udev 243.8-1.fc31 1.3 M
userspace-rcu 0.11.1-2.fc31 104 k
xz 5.2.4-6.fc31 147 k
Installing weak dependenc
kbd 2.0.4-14.fc31 391 k
kpartx 0.8.0-3.fc31 55 k
pigz 2.4-5.fc31 78 k
systemd-bootchart 233-5.fc31 71 k
Transaction Summary
========================================================
Install 32 Packages
Total size: 13 M
Total download size: 13 M
Installed size: 48 M
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v1 1/3] configure/Makefile: engine LIBS consistency
2020-06-28 19:27 [PATCH v1 0/3] fio: option to make engines load dynamically Yigal Korman
@ 2020-06-28 19:27 ` Yigal Korman
2020-06-28 19:27 ` [PATCH v1 2/3] configure: new --dynamic-libengines build option Yigal Korman
` (2 subsequent siblings)
3 siblings, 0 replies; 16+ messages in thread
From: Yigal Korman @ 2020-06-28 19:27 UTC (permalink / raw)
To: axboe; +Cc: Yigal Korman, fio
Some engines were adding their LIBS in the Makefile and some were
declaring them in the configure script.
This is inconsistent and confusing and prevents easy bulk changes to the
engines.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
---
Makefile | 14 ++++++++++++++
configure | 12 ------------
2 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/Makefile b/Makefile
index 7eb5e899..bf7f8de0 100644
--- a/Makefile
+++ b/Makefile
@@ -79,9 +79,15 @@ ifdef CONFIG_32BIT
endif
ifdef CONFIG_LIBAIO
SOURCE += engines/libaio.c
+ ifdef CONFIG_LIBAIO_URING
+ LIBS += -luring
+ else
+ LIBS += -laio
+ endif
endif
ifdef CONFIG_RDMA
SOURCE += engines/rdma.c
+ LIBS += -libverbs -lrdmacm
endif
ifdef CONFIG_POSIXAIO
SOURCE += engines/posixaio.c
@@ -106,12 +112,15 @@ ifdef CONFIG_WINDOWSAIO
endif
ifdef CONFIG_RADOS
SOURCE += engines/rados.c
+ LIBS += -lrados
endif
ifdef CONFIG_RBD
SOURCE += engines/rbd.c
+ LIBS += -lrbd -lrados
endif
ifdef CONFIG_HTTP
SOURCE += engines/http.c
+ LIBS += -lcurl -lssl -lcrypto
endif
SOURCE += oslib/asprintf.c
ifndef CONFIG_STRSEP
@@ -139,6 +148,7 @@ ifdef CONFIG_GFAPI
SOURCE += engines/glusterfs.c
SOURCE += engines/glusterfs_sync.c
SOURCE += engines/glusterfs_async.c
+ LIBS += -lgfapi -lglusterfs
ifdef CONFIG_GF_FADVISE
CFLAGS := "-DGFAPI_USE_FADVISE" $(CFLAGS)
endif
@@ -150,18 +160,22 @@ ifdef CONFIG_MTD
endif
ifdef CONFIG_PMEMBLK
SOURCE += engines/pmemblk.c
+ LIBS += -lpmemblk
endif
ifdef CONFIG_LINUX_DEVDAX
SOURCE += engines/dev-dax.c
+ LIBS += -lpmem
endif
ifdef CONFIG_LIBPMEM
SOURCE += engines/libpmem.c
+ LIBS += -lpmem
endif
ifdef CONFIG_IME
SOURCE += engines/ime.c
endif
ifdef CONFIG_LIBZBC
SOURCE += engines/libzbc.c
+ LIBS += -lzbc
endif
ifeq ($(CONFIG_TARGET_OS), Linux)
diff --git a/configure b/configure
index 63b30555..5d475bf3 100755
--- a/configure
+++ b/configure
@@ -605,11 +605,9 @@ int main(void)
EOF
if test "$libaio_uring" = "yes" && compile_prog "" "-luring" "libaio io_uring" ; then
libaio=yes
- LIBS="-luring $LIBS"
elif compile_prog "" "-laio" "libaio" ; then
libaio=yes
libaio_uring=no
- LIBS="-laio $LIBS"
else
if test "$libaio" = "yes" ; then
feature_not_found "linux AIO" "libaio-dev or libaio-devel"
@@ -859,7 +857,6 @@ int main(int argc, char **argv)
EOF
if test "$disable_rdma" != "yes" && compile_prog "" "-libverbs" "libverbs" ; then
libverbs="yes"
- LIBS="-libverbs $LIBS"
fi
print_config "libverbs" "$libverbs"
@@ -879,7 +876,6 @@ int main(int argc, char **argv)
EOF
if test "$disable_rdma" != "yes" && compile_prog "" "-lrdmacm" "rdma"; then
rdmacm="yes"
- LIBS="-lrdmacm $LIBS"
fi
print_config "rdmacm" "$rdmacm"
@@ -1770,10 +1766,8 @@ if test "$disable_http" != "yes"; then
if compile_prog "" "$HTTP_LIBS" "curl-new-ssl"; then
output_sym "CONFIG_HAVE_OPAQUE_HMAC_CTX"
http="yes"
- LIBS="$HTTP_LIBS $LIBS"
elif mv $TMPC2 $TMPC && compile_prog "" "$HTTP_LIBS" "curl-old-ssl"; then
http="yes"
- LIBS="$HTTP_LIBS $LIBS"
fi
fi
print_config "http engine" "$http"
@@ -1802,7 +1796,6 @@ int main(int argc, char **argv)
}
EOF
if test "$disable_rados" != "yes" && compile_prog "" "-lrados" "rados"; then
- LIBS="-lrados $LIBS"
rados="yes"
fi
print_config "Rados engine" "$rados"
@@ -1833,7 +1826,6 @@ int main(int argc, char **argv)
}
EOF
if test "$disable_rbd" != "yes" && compile_prog "" "-lrbd -lrados" "rbd"; then
- LIBS="-lrbd -lrados $LIBS"
rbd="yes"
fi
print_config "Rados Block Device engine" "$rbd"
@@ -1924,7 +1916,6 @@ int main(int argc, char **argv)
}
EOF
if test "$disable_gfapi" != "yes" && compile_prog "" "-lgfapi -lglusterfs" "gfapi"; then
- LIBS="-lgfapi -lglusterfs $LIBS"
gfapi="yes"
fi
print_config "Gluster API engine" "$gfapi"
@@ -2086,7 +2077,6 @@ int main(int argc, char **argv)
EOF
if compile_prog "" "-lpmem" "libpmem"; then
libpmem="yes"
- LIBS="-lpmem $LIBS"
fi
print_config "libpmem" "$libpmem"
@@ -2108,7 +2098,6 @@ int main(int argc, char **argv)
EOF
if compile_prog "" "-lpmemblk" "libpmemblk"; then
libpmemblk="yes"
- LIBS="-lpmemblk $LIBS"
fi
fi
print_config "libpmemblk" "$libpmemblk"
@@ -2432,7 +2421,6 @@ if compile_prog "" "-lzbc" "libzbc"; then
libzbcvermaj=$(pkg-config --modversion libzbc | sed 's/\.[0-9]*\.[0-9]*//')
if test "$libzbcvermaj" -ge "5" ; then
libzbc="yes"
- LIBS="-lzbc $LIBS"
else
print_config "libzbc engine" "Unsupported libzbc version (version 5 or above required)"
libzbc="no"
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-28 19:27 [PATCH v1 0/3] fio: option to make engines load dynamically Yigal Korman
2020-06-28 19:27 ` [PATCH v1 1/3] configure/Makefile: engine LIBS consistency Yigal Korman
@ 2020-06-28 19:27 ` Yigal Korman
2020-06-29 21:07 ` Jens Axboe
2020-06-28 19:27 ` [PATCH v1 3/3] fio: suggest installing engine package when missing Yigal Korman
2020-06-29 15:27 ` [PATCH v1 0/3] fio: option to make engines load dynamically Jeff Moyer
3 siblings, 1 reply; 16+ messages in thread
From: Yigal Korman @ 2020-06-28 19:27 UTC (permalink / raw)
To: axboe; +Cc: Yigal Korman, fio
When enabled, some of the more dependency-heavy internal engines are
converted to "plugin" engines, i.e. they are built into separate object
files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not
force a long list of package dependencies from the base fio package.
Signed-off-by: Yigal Korman <ykorman@gmail.com>
---
.gitignore | 1 +
Makefile | 92 ++++++++++++++++++++++++++++++++--------------
configure | 8 ++++
engines/dev-dax.c | 2 +-
engines/guasi.c | 2 +-
engines/http.c | 2 +-
engines/libaio.c | 2 +-
engines/libhdfs.c | 6 +--
engines/libiscsi.c | 6 +--
engines/libpmem.c | 2 +-
engines/libzbc.c | 2 +-
engines/nbd.c | 2 +-
engines/pmemblk.c | 2 +-
engines/rados.c | 2 +-
engines/rbd.c | 2 +-
engines/rdma.c | 9 +++--
ioengines.c | 20 +++++++++-
ioengines.h | 6 +++
os/os-linux.h | 2 +
19 files changed, 121 insertions(+), 49 deletions(-)
diff --git a/.gitignore b/.gitignore
index b84b0fda..0aa4a361 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
*.d
*.o
+*.so
*.exe
/.depend
/FIO-VERSION-FILE
diff --git a/Makefile b/Makefile
index bf7f8de0..3255389e 100644
--- a/Makefile
+++ b/Makefile
@@ -60,15 +60,17 @@ ifdef CONFIG_LIBHDFS
endif
ifdef CONFIG_LIBISCSI
- CFLAGS := $(LIBISCSI_CFLAGS) $(CFLAGS)
- LIBS += $(LIBISCSI_LIBS)
- SOURCE += engines/libiscsi.c
+ iscsi_SRCS = engines/libiscsi.c
+ iscsi_LIBS = $(LIBISCSI_LIBS)
+ iscsi_CFLAGS = $(LIBISCSI_LIBS)
+ ENGINES += iscsi
endif
ifdef CONFIG_LIBNBD
- CFLAGS := $(LIBNBD_CFLAGS) $(CFLAGS)
- LIBS += $(LIBNBD_LIBS)
- SOURCE += engines/nbd.c
+ nbd_SRCS = engines/nbd.c
+ nbd_LIBS = $(LIBNBD_LIBS)
+ nbd_CFLAGS = $(LIBNBD_CFLAGS)
+ ENGINES += nbd
endif
ifdef CONFIG_64BIT
@@ -78,16 +80,19 @@ ifdef CONFIG_32BIT
CFLAGS := -DBITS_PER_LONG=32 $(CFLAGS)
endif
ifdef CONFIG_LIBAIO
- SOURCE += engines/libaio.c
+ aio_SRCS = engines/libaio.c
+ aio_LIBS = -laio
ifdef CONFIG_LIBAIO_URING
- LIBS += -luring
+ aio_LIBS = -luring
else
- LIBS += -laio
+ aio_LIBS = -laio
endif
+ ENGINES += aio
endif
ifdef CONFIG_RDMA
- SOURCE += engines/rdma.c
- LIBS += -libverbs -lrdmacm
+ rdma_SRCS = engines/rdma.c
+ rdma_LIBS = -libverbs -lrdmacm
+ ENGINES += rdma
endif
ifdef CONFIG_POSIXAIO
SOURCE += engines/posixaio.c
@@ -102,7 +107,8 @@ ifdef CONFIG_LINUX_SPLICE
SOURCE += engines/splice.c
endif
ifdef CONFIG_GUASI
- SOURCE += engines/guasi.c
+ guasi_SRCS = engines/guasi.c
+ ENGINES += guasi
endif
ifdef CONFIG_SOLARISAIO
SOURCE += engines/solarisaio.c
@@ -111,16 +117,19 @@ ifdef CONFIG_WINDOWSAIO
SOURCE += engines/windowsaio.c
endif
ifdef CONFIG_RADOS
- SOURCE += engines/rados.c
- LIBS += -lrados
+ rados_SRCS = engines/rados.c
+ rados_LIBS = -lrados
+ ENGINES += rados
endif
ifdef CONFIG_RBD
- SOURCE += engines/rbd.c
- LIBS += -lrbd -lrados
+ rbd_SRCS = engines/rbd.c
+ rbd_LIBS = -lrbd -lrados
+ ENGINES += rbd
endif
ifdef CONFIG_HTTP
- SOURCE += engines/http.c
- LIBS += -lcurl -lssl -lcrypto
+ http_SRCS = engines/http.c
+ http_LIBS = -lcurl -lssl -lcrypto
+ ENGINES += http
endif
SOURCE += oslib/asprintf.c
ifndef CONFIG_STRSEP
@@ -159,23 +168,27 @@ ifdef CONFIG_MTD
SOURCE += oslib/libmtd_legacy.c
endif
ifdef CONFIG_PMEMBLK
- SOURCE += engines/pmemblk.c
- LIBS += -lpmemblk
+ pmemblk_SRCS = engines/pmemblk.c
+ pmemblk_LIBS = -lpmemblk
+ ENGINES += pmemblk
endif
ifdef CONFIG_LINUX_DEVDAX
- SOURCE += engines/dev-dax.c
- LIBS += -lpmem
+ devdax_SRCS = engines/dev-dax.c
+ devdax_LIBS = -lpmem
+ ENGINES += dev-dax
endif
ifdef CONFIG_LIBPMEM
- SOURCE += engines/libpmem.c
- LIBS += -lpmem
+ pmem_SRCS = engines/libpmem.c
+ pmem_LIBS = -lpmem
+ ENGINES += pmem
endif
ifdef CONFIG_IME
SOURCE += engines/ime.c
endif
ifdef CONFIG_LIBZBC
- SOURCE += engines/libzbc.c
- LIBS += -lzbc
+ zbc_SRCS = engines/libzbc.c
+ zbc_LIBS = -lzbc
+ ENGINES += zbc
endif
ifeq ($(CONFIG_TARGET_OS), Linux)
@@ -237,6 +250,26 @@ ifneq (,$(findstring CYGWIN,$(CONFIG_TARGET_OS)))
CFLAGS := -DPSAPI_VERSION=1 -Ios/windows/posix/include -Wno-format $(CFLAGS)
endif
+ifdef CONFIG_DYNAMIC_ENGINES
+ DYNAMIC_ENGS := $(ENGINES)
+define engine_template =
+$(1)_OBJS := $$($(1)_SRCS:.c=.o)
+$$($(1)_OBJS): CFLAGS := -fPIC $$($(1)_CFLAGS) $(CFLAGS)
+engines/lib$(1).so: $$($(1)_OBJS)
+ $$(QUIET_LINK)$(CC) -shared -rdynamic -fPIC -Wl,-soname,lib$(1).so.1 $$($(1)_LIBS) -o $$@ $$<
+ENGS_OBJS += engines/lib$(1).so
+all install: $(ENGS_OBJS)
+endef
+else # !CONFIG_DYNAMIC_ENGINES
+define engine_template =
+SOURCE += $$($(1)_SRCS)
+LIBS += $$($(1)_LIBS)
+CFLAGS := $$($(1)_CFLAGS) $(CFLAGS)
+endef
+endif
+
+$(foreach eng,$(ENGINES),$(eval $(call engine_template,$(eng))))
+
OBJS := $(SOURCE:.c=.o)
FIO_OBJS = $(OBJS) fio.o
@@ -386,6 +419,7 @@ else
endif
prefix = $(INSTALL_PREFIX)
bindir = $(prefix)/bin
+libdir = $(prefix)/lib/fio
ifeq ($(CONFIG_TARGET_OS), Darwin)
mandir = /usr/share/man
@@ -534,7 +568,7 @@ unittests/unittest: $(UT_OBJS) $(UT_TARGET_OBJS)
endif
clean: FORCE
- @rm -f .depend $(FIO_OBJS) $(GFIO_OBJS) $(OBJS) $(T_OBJS) $(UT_OBJS) $(PROGS) $(T_PROGS) $(T_TEST_PROGS) core.* core gfio unittests/unittest FIO-VERSION-FILE *.[do] lib/*.d oslib/*.[do] crc/*.d engines/*.[do] profiles/*.[do] t/*.[do] unittests/*.[do] unittests/*/*.[do] config-host.mak config-host.h y.tab.[ch] lex.yy.c exp/*.[do] lexer.h
+ @rm -f .depend $(FIO_OBJS) $(GFIO_OBJS) $(OBJS) $(ENGS_OBJS) $(T_OBJS) $(UT_OBJS) $(PROGS) $(T_PROGS) $(T_TEST_PROGS) core.* core gfio unittests/unittest FIO-VERSION-FILE *.[do] lib/*.d oslib/*.[do] crc/*.d engines/*.[do] profiles/*.[do] t/*.[do] unittests/*.[do] unittests/*/*.[do] config-host.mak config-host.h y.tab.[ch] lex.yy.c exp/*.[do] lexer.h
@rm -f t/fio-btrace2fio t/io_uring t/read-to-pipe-async
@rm -rf doc/output
@@ -574,6 +608,10 @@ fulltest:
install: $(PROGS) $(SCRIPTS) tools/plot/fio2gnuplot.1 FORCE
$(INSTALL) -m 755 -d $(DESTDIR)$(bindir)
$(INSTALL) $(PROGS) $(SCRIPTS) $(DESTDIR)$(bindir)
+ifdef CONFIG_DYNAMIC_ENGINES
+ $(INSTALL) -m 755 -d $(DESTDIR)$(libdir)
+ $(INSTALL) -m 755 $(SRCDIR)/engines/*.so $(DESTDIR)$(libdir)
+endif
$(INSTALL) -m 755 -d $(DESTDIR)$(mandir)/man1
$(INSTALL) -m 644 $(SRCDIR)/fio.1 $(DESTDIR)$(mandir)/man1
$(INSTALL) -m 644 $(SRCDIR)/tools/fio_generate_plots.1 $(DESTDIR)$(mandir)/man1
diff --git a/configure b/configure
index 5d475bf3..0e8c27b1 100755
--- a/configure
+++ b/configure
@@ -151,6 +151,7 @@ march_set="no"
libiscsi="no"
libnbd="no"
libaio_uring="no"
+dynamic_engines="no"
prefix=/usr/local
# parse options
@@ -215,6 +216,8 @@ for opt do
;;
--enable-libaio-uring) libaio_uring="yes"
;;
+ --dynamic-libengines) dynamic_engines="yes"
+ ;;
--help)
show_help="yes"
;;
@@ -254,6 +257,7 @@ if test "$show_help" = "yes" ; then
echo "--enable-libnbd Enable libnbd (NBD engine) support"
echo "--disable-tcmalloc Disable tcmalloc support"
echo "--enable-libaio-uring Enable libaio emulated over io_uring"
+ echo "--dynamic-libengines Lib-based ioengines as dynamic libraries"
exit $exit_val
fi
@@ -2954,6 +2958,10 @@ if test "$libnbd" = "yes" ; then
echo "LIBNBD_CFLAGS=$libnbd_cflags" >> $config_host_mak
echo "LIBNBD_LIBS=$libnbd_libs" >> $config_host_mak
fi
+if test "$dynamic_engines" = "yes" ; then
+ output_sym "CONFIG_DYNAMIC_ENGINES"
+fi
+print_config "Lib-based ioengines dynamic" "$dynamic_engines"
cat > $TMPC << EOF
int main(int argc, char **argv)
{
diff --git a/engines/dev-dax.c b/engines/dev-dax.c
index 422ea634..1d0f66cb 100644
--- a/engines/dev-dax.c
+++ b/engines/dev-dax.c
@@ -328,7 +328,7 @@ fio_devdax_get_file_size(struct thread_data *td, struct fio_file *f)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "dev-dax",
.version = FIO_IOOPS_VERSION,
.init = fio_devdax_init,
diff --git a/engines/guasi.c b/engines/guasi.c
index cb26802c..d4121757 100644
--- a/engines/guasi.c
+++ b/engines/guasi.c
@@ -242,7 +242,7 @@ static int fio_guasi_init(struct thread_data *td)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "guasi",
.version = FIO_IOOPS_VERSION,
.init = fio_guasi_init,
diff --git a/engines/http.c b/engines/http.c
index 275fcab5..7a61b132 100644
--- a/engines/http.c
+++ b/engines/http.c
@@ -639,7 +639,7 @@ static int fio_http_invalidate(struct thread_data *td, struct fio_file *f)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "http",
.version = FIO_IOOPS_VERSION,
.flags = FIO_DISKLESSIO | FIO_SYNCIO,
diff --git a/engines/libaio.c b/engines/libaio.c
index 398fdf91..b909b79e 100644
--- a/engines/libaio.c
+++ b/engines/libaio.c
@@ -445,7 +445,7 @@ static int fio_libaio_init(struct thread_data *td)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "libaio",
.version = FIO_IOOPS_VERSION,
.flags = FIO_ASYNCIO_SYNC_TRIM,
diff --git a/engines/libhdfs.c b/engines/libhdfs.c
index c57fcea6..9ca82f78 100644
--- a/engines/libhdfs.c
+++ b/engines/libhdfs.c
@@ -393,7 +393,7 @@ static void fio_hdfsio_io_u_free(struct thread_data *td, struct io_u *io_u)
}
}
-static struct ioengine_ops ioengine_hdfs = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "libhdfs",
.version = FIO_IOOPS_VERSION,
.flags = FIO_SYNCIO | FIO_DISKLESSIO | FIO_NODISKUTIL,
@@ -412,10 +412,10 @@ static struct ioengine_ops ioengine_hdfs = {
static void fio_init fio_hdfsio_register(void)
{
- register_ioengine(&ioengine_hdfs);
+ register_ioengine(&ioengine);
}
static void fio_exit fio_hdfsio_unregister(void)
{
- unregister_ioengine(&ioengine_hdfs);
+ unregister_ioengine(&ioengine);
}
diff --git a/engines/libiscsi.c b/engines/libiscsi.c
index 35761a61..c97b5709 100644
--- a/engines/libiscsi.c
+++ b/engines/libiscsi.c
@@ -383,7 +383,7 @@ static struct io_u *fio_iscsi_event(struct thread_data *td, int event)
return io_u;
}
-static struct ioengine_ops ioengine_iscsi = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "libiscsi",
.version = FIO_IOOPS_VERSION,
.flags = FIO_SYNCIO | FIO_DISKLESSIO | FIO_NODISKUTIL,
@@ -402,10 +402,10 @@ static struct ioengine_ops ioengine_iscsi = {
static void fio_init fio_iscsi_register(void)
{
- register_ioengine(&ioengine_iscsi);
+ register_ioengine(&ioengine);
}
static void fio_exit fio_iscsi_unregister(void)
{
- unregister_ioengine(&ioengine_iscsi);
+ unregister_ioengine(&ioengine);
}
diff --git a/engines/libpmem.c b/engines/libpmem.c
index 99c7b50d..3f63055c 100644
--- a/engines/libpmem.c
+++ b/engines/libpmem.c
@@ -558,7 +558,7 @@ static int fio_libpmem_close_file(struct thread_data *td, struct fio_file *f)
return generic_close_file(td, f);
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "libpmem",
.version = FIO_IOOPS_VERSION,
.init = fio_libpmem_init,
diff --git a/engines/libzbc.c b/engines/libzbc.c
index 9e568334..fdde8ca6 100644
--- a/engines/libzbc.c
+++ b/engines/libzbc.c
@@ -397,7 +397,7 @@ static enum fio_q_status libzbc_queue(struct thread_data *td, struct io_u *io_u)
return FIO_Q_COMPLETED;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "libzbc",
.version = FIO_IOOPS_VERSION,
.open_file = libzbc_open_file,
diff --git a/engines/nbd.c b/engines/nbd.c
index 53237929..b0ba75e6 100644
--- a/engines/nbd.c
+++ b/engines/nbd.c
@@ -328,7 +328,7 @@ static int nbd_invalidate(struct thread_data *td, struct fio_file *f)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "nbd",
.version = FIO_IOOPS_VERSION,
.options = options,
diff --git a/engines/pmemblk.c b/engines/pmemblk.c
index 730f4d77..bd837655 100644
--- a/engines/pmemblk.c
+++ b/engines/pmemblk.c
@@ -426,7 +426,7 @@ static int fio_pmemblk_unlink_file(struct thread_data *td, struct fio_file *f)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "pmemblk",
.version = FIO_IOOPS_VERSION,
.queue = fio_pmemblk_queue,
diff --git a/engines/rados.c b/engines/rados.c
index d4413427..42ee48ff 100644
--- a/engines/rados.c
+++ b/engines/rados.c
@@ -444,7 +444,7 @@ static int fio_rados_io_u_init(struct thread_data *td, struct io_u *io_u)
}
/* ioengine_ops for get_ioengine() */
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "rados",
.version = FIO_IOOPS_VERSION,
.flags = FIO_DISKLESSIO,
diff --git a/engines/rbd.c b/engines/rbd.c
index a08f4775..268b6ebd 100644
--- a/engines/rbd.c
+++ b/engines/rbd.c
@@ -668,7 +668,7 @@ static int fio_rbd_io_u_init(struct thread_data *td, struct io_u *io_u)
return 0;
}
-static struct ioengine_ops ioengine = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "rbd",
.version = FIO_IOOPS_VERSION,
.setup = fio_rbd_setup,
diff --git a/engines/rdma.c b/engines/rdma.c
index f192f432..f4471869 100644
--- a/engines/rdma.c
+++ b/engines/rdma.c
@@ -226,7 +226,8 @@ static int client_recv(struct thread_data *td, struct ibv_wc *wc)
rd->rmt_nr = ntohl(rd->recv_buf.nr);
for (i = 0; i < rd->rmt_nr; i++) {
- rd->rmt_us[i].buf = be64_to_cpu(rd->recv_buf.rmt_us[i].buf);
+ rd->rmt_us[i].buf = __be64_to_cpu(
+ rd->recv_buf.rmt_us[i].buf);
rd->rmt_us[i].rkey = ntohl(rd->recv_buf.rmt_us[i].rkey);
rd->rmt_us[i].size = ntohl(rd->recv_buf.rmt_us[i].size);
@@ -1389,7 +1390,7 @@ static int fio_rdmaio_setup(struct thread_data *td)
return 0;
}
-static struct ioengine_ops ioengine_rw = {
+FIO_STATIC struct ioengine_ops ioengine = {
.name = "rdma",
.version = FIO_IOOPS_VERSION,
.setup = fio_rdmaio_setup,
@@ -1410,10 +1411,10 @@ static struct ioengine_ops ioengine_rw = {
static void fio_init fio_rdmaio_register(void)
{
- register_ioengine(&ioengine_rw);
+ register_ioengine(&ioengine);
}
static void fio_exit fio_rdmaio_unregister(void)
{
- unregister_ioengine(&ioengine_rw);
+ unregister_ioengine(&ioengine);
}
diff --git a/ioengines.c b/ioengines.c
index 2c7a0df9..78262bc1 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -75,6 +75,19 @@ static struct ioengine_ops *find_ioengine(const char *name)
return NULL;
}
+#ifdef CONFIG_DYNAMIC_ENGINES
+static void *dlopen_external(struct thread_data *td, const char *engine)
+{
+ char engine_path[PATH_MAX];
+
+ sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
+
+ return dlopen(engine_path, RTLD_LAZY);
+}
+#else
+#define dlopen_external(td, engine) (NULL)
+#endif
+
static struct ioengine_ops *dlopen_ioengine(struct thread_data *td,
const char *engine_lib)
{
@@ -86,8 +99,11 @@ static struct ioengine_ops *dlopen_ioengine(struct thread_data *td,
dlerror();
dlhandle = dlopen(engine_lib, RTLD_LAZY);
if (!dlhandle) {
- td_vmsg(td, -1, dlerror(), "dlopen");
- return NULL;
+ dlhandle = dlopen_external(td, engine_lib);
+ if (!dlhandle) {
+ td_vmsg(td, -1, dlerror(), "dlopen");
+ return NULL;
+ }
}
/*
diff --git a/ioengines.h b/ioengines.h
index f48b4db9..54dadba2 100644
--- a/ioengines.h
+++ b/ioengines.h
@@ -10,6 +10,12 @@
#define FIO_IOOPS_VERSION 26
+#ifndef CONFIG_DYNAMIC_ENGINES
+#define FIO_STATIC static
+#else
+#define FIO_STATIC
+#endif
+
/*
* io_ops->queue() return values
*/
diff --git a/os/os-linux.h b/os/os-linux.h
index 6ec7243d..65d3b429 100644
--- a/os/os-linux.h
+++ b/os/os-linux.h
@@ -58,6 +58,8 @@
#define OS_MAP_ANON MAP_ANONYMOUS
+#define FIO_EXT_ENG_DIR "/usr/lib/fio"
+
typedef cpu_set_t os_cpu_mask_t;
#ifdef CONFIG_3ARG_AFFINITY
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v1 3/3] fio: suggest installing engine package when missing
2020-06-28 19:27 [PATCH v1 0/3] fio: option to make engines load dynamically Yigal Korman
2020-06-28 19:27 ` [PATCH v1 1/3] configure/Makefile: engine LIBS consistency Yigal Korman
2020-06-28 19:27 ` [PATCH v1 2/3] configure: new --dynamic-libengines build option Yigal Korman
@ 2020-06-28 19:27 ` Yigal Korman
2020-06-29 15:25 ` Jeff Moyer
2020-06-29 15:27 ` [PATCH v1 0/3] fio: option to make engines load dynamically Jeff Moyer
3 siblings, 1 reply; 16+ messages in thread
From: Yigal Korman @ 2020-06-28 19:27 UTC (permalink / raw)
To: axboe; +Cc: Yigal Korman, fio
Signed-off-by: Yigal Korman <ykorman@gmail.com>
---
ioengines.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/ioengines.c b/ioengines.c
index 78262bc1..20224e59 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
static void *dlopen_external(struct thread_data *td, const char *engine)
{
char engine_path[PATH_MAX];
+ void *dlhandle;
sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
- return dlopen(engine_path, RTLD_LAZY);
+ dlhandle = dlopen(engine_path, RTLD_LAZY);
+ if (!dlhandle)
+ log_info("Engine not available; Install the fio-engine-%s package\n",
+ engine);
+
+ return dlhandle;
}
#else
#define dlopen_external(td, engine) (NULL)
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v1 3/3] fio: suggest installing engine package when missing
2020-06-28 19:27 ` [PATCH v1 3/3] fio: suggest installing engine package when missing Yigal Korman
@ 2020-06-29 15:25 ` Jeff Moyer
2020-06-30 19:29 ` Yigal Korman
0 siblings, 1 reply; 16+ messages in thread
From: Jeff Moyer @ 2020-06-29 15:25 UTC (permalink / raw)
To: Yigal Korman; +Cc: axboe, fio
Yigal Korman <ykorman@gmail.com> writes:
> Signed-off-by: Yigal Korman <ykorman@gmail.com>
> ---
> ioengines.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/ioengines.c b/ioengines.c
> index 78262bc1..20224e59 100644
> --- a/ioengines.c
> +++ b/ioengines.c
> @@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
> static void *dlopen_external(struct thread_data *td, const char *engine)
> {
> char engine_path[PATH_MAX];
> + void *dlhandle;
>
> sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
>
> - return dlopen(engine_path, RTLD_LAZY);
> + dlhandle = dlopen(engine_path, RTLD_LAZY);
> + if (!dlhandle)
> + log_info("Engine not available; Install the fio-engine-%s package\n",
> + engine);
> +
> + return dlhandle;
> }
> #else
> #define dlopen_external(td, engine) (NULL)
This assumes fio is part of a package, and that's not always (often?)
the case. It may be that the engine simply wasn't built.
-Jeff
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 0/3] fio: option to make engines load dynamically
2020-06-28 19:27 [PATCH v1 0/3] fio: option to make engines load dynamically Yigal Korman
` (2 preceding siblings ...)
2020-06-28 19:27 ` [PATCH v1 3/3] fio: suggest installing engine package when missing Yigal Korman
@ 2020-06-29 15:27 ` Jeff Moyer
2020-06-29 18:44 ` Yigal Korman
2020-06-29 18:47 ` Yigal Korman
3 siblings, 2 replies; 16+ messages in thread
From: Jeff Moyer @ 2020-06-29 15:27 UTC (permalink / raw)
To: Yigal Korman; +Cc: axboe, fio
Yigal Korman <ykorman@gmail.com> writes:
> Hi,
>
> The following patchset provides a configuration option that converts a
> select set of engines into external engines that are loaded dynamically
> on demand. The purpose of this option is to provide distribution
> packagers the ability to separate the core fio functionality from the
> addition optional engines. Some of the optional engines have multiple
> dependencies that increase the footprint of fio considerably even when
> the user requires only the basic 'sync' engine. See examples of the
> issue below.
Did you forget to include the examples?
I'm not sure I see a benefit to this change. fio isn't some core system
service--it's a benchmarking tool. Maybe your examples shed more light
on why this matters?
-Jeff
>
> I've tried to keep the changes as small and unobtrusive as possible.
> The only significant change is the way an engine is defined in the
> Makefile (variables are now prefixed with engine name). If the option
> is provided, the Makefile will build the engines as external libraries
> and install them in /usr/lib/fio by default. Fio will then search for
> the engines in that location if the engine is not compiled in. The last
> patch should help with the migration to the new scheme by suggesting
> that an additional package is required if the engine is not found.
>
> Future work to consider:
> * Move the engine help text from fio to the engines themselves. Create
> * a whitelist of engines so that only engines that are part of the main
> * codebase are searched and loaded automatically. Convert more engines
> * to this scheme. The current set of engines was selected based on
> * simplicity and external dependencies. Engines with multiple exported
> * flavors (i.e. glusterfs) are not converted yet.
>
> Branch is also available on github, at
> https://github.com/ykorman/fio/tree/dynamic-libengines.
>
> A sample of how to package fio with this new mode for RPM-based
> distributions is available here:
> https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines
>
> A sample of the packages for Fedora can be found here:
> https://copr.fedorainfracloud.org/coprs/ykorman/fio/
>
> See examples of how this improves our quality of life (tm) below the
> pull summary.
>
>
> Yigal Korman (3):
> configure/Makefile: engine LIBS consistency
> configure: new --dynamic-libengines build option
> fio: suggest installing engine package when missing
>
> .gitignore | 1 +
> Makefile | 86 +++++++++++++++++++++++++++++++++++++---------
> configure | 20 +++++------
> engines/dev-dax.c | 2 +-
> engines/guasi.c | 2 +-
> engines/http.c | 2 +-
> engines/libaio.c | 2 +-
> engines/libhdfs.c | 6 ++--
> engines/libiscsi.c | 6 ++--
> engines/libpmem.c | 2 +-
> engines/libzbc.c | 2 +-
> engines/nbd.c | 2 +-
> engines/pmemblk.c | 2 +-
> engines/rados.c | 2 +-
> engines/rbd.c | 2 +-
> engines/rdma.c | 9 ++---
> ioengines.c | 26 ++++++++++++--
> ioengines.h | 6 ++++
> os/os-linux.h | 2 ++
> 19 files changed, 131 insertions(+), 51 deletions(-)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 0/3] fio: option to make engines load dynamically
2020-06-29 15:27 ` [PATCH v1 0/3] fio: option to make engines load dynamically Jeff Moyer
@ 2020-06-29 18:44 ` Yigal Korman
2020-06-29 18:47 ` Yigal Korman
1 sibling, 0 replies; 16+ messages in thread
From: Yigal Korman @ 2020-06-29 18:44 UTC (permalink / raw)
To: Jeff Moyer; +Cc: axboe, fio
[-- Attachment #1: Type: text/plain, Size: 4493 bytes --]
Hi Jeff,
Thanks for taking a look at this.
Answers inline below.
On Mon, Jun 29, 2020 at 6:28 PM Jeff Moyer <jmoyer@redhat.com> wrote:
> Yigal Korman <ykorman@gmail.com> writes:
>
> > Hi,
> >
> > The following patchset provides a configuration option that converts a
> > select set of engines into external engines that are loaded dynamically
> > on demand. The purpose of this option is to provide distribution
> > packagers the ability to separate the core fio functionality from the
> > addition optional engines. Some of the optional engines have multiple
> > dependencies that increase the footprint of fio considerably even when
> > the user requires only the basic 'sync' engine. See examples of the
> > issue below.
>
> Did you forget to include the examples?
>
Sorry, I didn't, it's easy to miss though, they're below the pull summary.
They are a bit verbose, so I'll recap here:
Without the patchset, on a Fedora 31 system, installing fio brings a total
of 38 package dependencies, the installed size is 57MB.
With the patchset, installing fio brings a single dependency (numactl-libs)
and the installed size is 2.1MB.
These numbers are relatively small but when one needs to bring fio to an
offline server, collecting these dependencies is a hassle.
Also, bundling fio into a container where the base image is small (Fedora
31 is ~200MB), the install size becomes more prominent.
> I'm not sure I see a benefit to this change. fio isn't some core system
> service--it's a benchmarking tool. Maybe your examples shed more light
> on why this matters?
>
I actually think that although fio is a benchmarking tool, it became so
ubiquitous in the storage world that I'd consider it a core tool.
I think all recent storage-related research papers and product brochures
that I saw have always included fio performance numbers.
Perhaps, if it will be possible to reduce its footprint, we might
start seeing it on the distro's base install.
Regards,
Yigal
> -Jeff
>
> >
> > I've tried to keep the changes as small and unobtrusive as possible.
> > The only significant change is the way an engine is defined in the
> > Makefile (variables are now prefixed with engine name). If the option
> > is provided, the Makefile will build the engines as external libraries
> > and install them in /usr/lib/fio by default. Fio will then search for
> > the engines in that location if the engine is not compiled in. The last
> > patch should help with the migration to the new scheme by suggesting
> > that an additional package is required if the engine is not found.
> >
> > Future work to consider:
> > * Move the engine help text from fio to the engines themselves. Create
> > * a whitelist of engines so that only engines that are part of the main
> > * codebase are searched and loaded automatically. Convert more engines
> > * to this scheme. The current set of engines was selected based on
> > * simplicity and external dependencies. Engines with multiple exported
> > * flavors (i.e. glusterfs) are not converted yet.
> >
> > Branch is also available on github, at
> > https://github.com/ykorman/fio/tree/dynamic-libengines.
> >
> > A sample of how to package fio with this new mode for RPM-based
> > distributions is available here:
> >
> https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines
> >
> > A sample of the packages for Fedora can be found here:
> > https://copr.fedorainfracloud.org/coprs/ykorman/fio/
> >
> > See examples of how this improves our quality of life (tm) below the
> > pull summary.
> >
> >
> > Yigal Korman (3):
> > configure/Makefile: engine LIBS consistency
> > configure: new --dynamic-libengines build option
> > fio: suggest installing engine package when missing
> >
> > .gitignore | 1 +
> > Makefile | 86 +++++++++++++++++++++++++++++++++++++---------
> > configure | 20 +++++------
> > engines/dev-dax.c | 2 +-
> > engines/guasi.c | 2 +-
> > engines/http.c | 2 +-
> > engines/libaio.c | 2 +-
> > engines/libhdfs.c | 6 ++--
> > engines/libiscsi.c | 6 ++--
> > engines/libpmem.c | 2 +-
> > engines/libzbc.c | 2 +-
> > engines/nbd.c | 2 +-
> > engines/pmemblk.c | 2 +-
> > engines/rados.c | 2 +-
> > engines/rbd.c | 2 +-
> > engines/rdma.c | 9 ++---
> > ioengines.c | 26 ++++++++++++--
> > ioengines.h | 6 ++++
> > os/os-linux.h | 2 ++
> > 19 files changed, 131 insertions(+), 51 deletions(-)
>
>
[-- Attachment #2: Type: text/html, Size: 6273 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 0/3] fio: option to make engines load dynamically
2020-06-29 15:27 ` [PATCH v1 0/3] fio: option to make engines load dynamically Jeff Moyer
2020-06-29 18:44 ` Yigal Korman
@ 2020-06-29 18:47 ` Yigal Korman
1 sibling, 0 replies; 16+ messages in thread
From: Yigal Korman @ 2020-06-29 18:47 UTC (permalink / raw)
To: Jeff Moyer; +Cc: axboe, fio
Hi Jeff,
Thanks for taking a look at this.
Answers (this time in plain text, sorry about that) inline below.
On Mon, Jun 29, 2020 at 6:28 PM Jeff Moyer <jmoyer@redhat.com> wrote:
>
> Yigal Korman <ykorman@gmail.com> writes:
>
> > Hi,
> >
> > The following patchset provides a configuration option that converts a
> > select set of engines into external engines that are loaded dynamically
> > on demand. The purpose of this option is to provide distribution
> > packagers the ability to separate the core fio functionality from the
> > addition optional engines. Some of the optional engines have multiple
> > dependencies that increase the footprint of fio considerably even when
> > the user requires only the basic 'sync' engine. See examples of the
> > issue below.
>
> Did you forget to include the examples?
Sorry, I didn't, it's easy to miss though, they're below the pull summary.
They are a bit verbose, so I'll recap here:
Without the patchset, on a Fedora 31 system, installing fio brings a
total of 38 package dependencies, the installed size is 57MB.
With the patchset, installing fio brings a single dependency
(numactl-libs) and the installed size is 2.1MB.
These numbers are relatively small but when one needs to bring fio to
an offline server, collecting these dependencies is a hassle.
Also, bundling fio into a container where the base image is small
(Fedora 31 is ~200MB), the install size becomes more prominent.
>
> I'm not sure I see a benefit to this change. fio isn't some core system
> service--it's a benchmarking tool. Maybe your examples shed more light
> on why this matters?
I actually think that although fio is a benchmarking tool, it became
so ubiquitous in the storage world that I'd consider it a core tool.
I think all recent storage-related research papers and product
brochures that I saw have always included fio performance numbers.
Perhaps, if it will be possible to reduce its footprint, we might
start seeing it on the distro's base install.
Regards,
Yigal
>
> -Jeff
>
> >
> > I've tried to keep the changes as small and unobtrusive as possible.
> > The only significant change is the way an engine is defined in the
> > Makefile (variables are now prefixed with engine name). If the option
> > is provided, the Makefile will build the engines as external libraries
> > and install them in /usr/lib/fio by default. Fio will then search for
> > the engines in that location if the engine is not compiled in. The last
> > patch should help with the migration to the new scheme by suggesting
> > that an additional package is required if the engine is not found.
> >
> > Future work to consider:
> > * Move the engine help text from fio to the engines themselves. Create
> > * a whitelist of engines so that only engines that are part of the main
> > * codebase are searched and loaded automatically. Convert more engines
> > * to this scheme. The current set of engines was selected based on
> > * simplicity and external dependencies. Engines with multiple exported
> > * flavors (i.e. glusterfs) are not converted yet.
> >
> > Branch is also available on github, at
> > https://github.com/ykorman/fio/tree/dynamic-libengines.
> >
> > A sample of how to package fio with this new mode for RPM-based
> > distributions is available here:
> > https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines
> >
> > A sample of the packages for Fedora can be found here:
> > https://copr.fedorainfracloud.org/coprs/ykorman/fio/
> >
> > See examples of how this improves our quality of life (tm) below the
> > pull summary.
> >
> >
> > Yigal Korman (3):
> > configure/Makefile: engine LIBS consistency
> > configure: new --dynamic-libengines build option
> > fio: suggest installing engine package when missing
> >
> > .gitignore | 1 +
> > Makefile | 86 +++++++++++++++++++++++++++++++++++++---------
> > configure | 20 +++++------
> > engines/dev-dax.c | 2 +-
> > engines/guasi.c | 2 +-
> > engines/http.c | 2 +-
> > engines/libaio.c | 2 +-
> > engines/libhdfs.c | 6 ++--
> > engines/libiscsi.c | 6 ++--
> > engines/libpmem.c | 2 +-
> > engines/libzbc.c | 2 +-
> > engines/nbd.c | 2 +-
> > engines/pmemblk.c | 2 +-
> > engines/rados.c | 2 +-
> > engines/rbd.c | 2 +-
> > engines/rdma.c | 9 ++---
> > ioengines.c | 26 ++++++++++++--
> > ioengines.h | 6 ++++
> > os/os-linux.h | 2 ++
> > 19 files changed, 131 insertions(+), 51 deletions(-)
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-28 19:27 ` [PATCH v1 2/3] configure: new --dynamic-libengines build option Yigal Korman
@ 2020-06-29 21:07 ` Jens Axboe
2020-06-30 19:25 ` Yigal Korman
0 siblings, 1 reply; 16+ messages in thread
From: Jens Axboe @ 2020-06-29 21:07 UTC (permalink / raw)
To: Yigal Korman; +Cc: fio
On 6/28/20 1:27 PM, Yigal Korman wrote:
> When enabled, some of the more dependency-heavy internal engines are
> converted to "plugin" engines, i.e. they are built into separate object
> files and are loaded by fio on demand.
> This helps downstream distros package these engines separately and not
> force a long list of package dependencies from the base fio package.
How does this impact the performance of the engine? It'd be interesting
to run a test with something ala:
fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
with the current build, then apply these patches (and turn the null engine
into an externally loadable engine, of course), and re-run the test case.
For what it's worth, I like the change in general, as dependencies do
pile on. But I'd like to ensure that we're not taking a performance hit
for something like this.
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-29 21:07 ` Jens Axboe
@ 2020-06-30 19:25 ` Yigal Korman
2020-06-30 19:35 ` Jens Axboe
0 siblings, 1 reply; 16+ messages in thread
From: Yigal Korman @ 2020-06-30 19:25 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
On Tue, Jun 30, 2020 at 12:07 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 6/28/20 1:27 PM, Yigal Korman wrote:
> > When enabled, some of the more dependency-heavy internal engines are
> > converted to "plugin" engines, i.e. they are built into separate object
> > files and are loaded by fio on demand.
> > This helps downstream distros package these engines separately and not
> > force a long list of package dependencies from the base fio package.
>
> How does this impact the performance of the engine? It'd be interesting
> to run a test with something ala:
>
> fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
>
> with the current build, then apply these patches (and turn the null engine
> into an externally loadable engine, of course), and re-run the test case.
>
> For what it's worth, I like the change in general, as dependencies do
> pile on. But I'd like to ensure that we're not taking a performance hit
> for something like this.
Great to hear.
Here are the results of the run you suggested:
current build (statically linked) -
[root@host fio]# ./fio.static --name=test --ioengine=null --size=100g
--rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
stdev=38.59, samples=52
iops : min=969293, max=1015596, avg=1000951.42,
stdev=9879.80, samples=52
cpu : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s
(4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec
With patches applied[0] -
[root@host fio]# ./fio --name=test --ioengine=null --size=100g
--rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
stdev=45.93, samples=52
iops : min=960962, max=1021038, avg=1007067.92,
stdev=11758.92, samples=52
cpu : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s
(4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec
I wasn't expecting dynamic linking to have a performance impact and
the results seem to agree.
Thanks!
[0] branch with the null engine converted to dynamic:
https://github.com/ykorman/fio/tree/devel
Yigal
>
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 3/3] fio: suggest installing engine package when missing
2020-06-29 15:25 ` Jeff Moyer
@ 2020-06-30 19:29 ` Yigal Korman
2020-06-30 19:41 ` Jens Axboe
0 siblings, 1 reply; 16+ messages in thread
From: Yigal Korman @ 2020-06-30 19:29 UTC (permalink / raw)
To: Jeff Moyer; +Cc: Jens Axboe, fio
On Mon, Jun 29, 2020 at 6:25 PM Jeff Moyer <jmoyer@redhat.com> wrote:
>
> Yigal Korman <ykorman@gmail.com> writes:
>
> > Signed-off-by: Yigal Korman <ykorman@gmail.com>
> > ---
> > ioengines.c | 8 +++++++-
> > 1 file changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/ioengines.c b/ioengines.c
> > index 78262bc1..20224e59 100644
> > --- a/ioengines.c
> > +++ b/ioengines.c
> > @@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
> > static void *dlopen_external(struct thread_data *td, const char *engine)
> > {
> > char engine_path[PATH_MAX];
> > + void *dlhandle;
> >
> > sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
> >
> > - return dlopen(engine_path, RTLD_LAZY);
> > + dlhandle = dlopen(engine_path, RTLD_LAZY);
> > + if (!dlhandle)
> > + log_info("Engine not available; Install the fio-engine-%s package\n",
> > + engine);
> > +
> > + return dlhandle;
> > }
> > #else
> > #define dlopen_external(td, engine) (NULL)
>
> This assumes fio is part of a package, and that's not always (often?)
> the case. It may be that the engine simply wasn't built.
I agree.
This patch is more of a showcase of how to help package maintainers
transition users to the new packaging model.
I will drop it from the next patchset version.
This patch might have made more sense if I had implemented a whitelist
of engines that were compiled dynamically.
Thanks,
Yigal
>
> -Jeff
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-30 19:25 ` Yigal Korman
@ 2020-06-30 19:35 ` Jens Axboe
2020-06-30 19:42 ` Jens Axboe
0 siblings, 1 reply; 16+ messages in thread
From: Jens Axboe @ 2020-06-30 19:35 UTC (permalink / raw)
To: Yigal Korman; +Cc: fio
On 6/30/20 1:25 PM, Yigal Korman wrote:
> On Tue, Jun 30, 2020 at 12:07 AM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 6/28/20 1:27 PM, Yigal Korman wrote:
>>> When enabled, some of the more dependency-heavy internal engines are
>>> converted to "plugin" engines, i.e. they are built into separate object
>>> files and are loaded by fio on demand.
>>> This helps downstream distros package these engines separately and not
>>> force a long list of package dependencies from the base fio package.
>>
>> How does this impact the performance of the engine? It'd be interesting
>> to run a test with something ala:
>>
>> fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
>>
>> with the current build, then apply these patches (and turn the null engine
>> into an externally loadable engine, of course), and re-run the test case.
>>
>> For what it's worth, I like the change in general, as dependencies do
>> pile on. But I'd like to ensure that we're not taking a performance hit
>> for something like this.
>
> Great to hear.
>
> Here are the results of the run you suggested:
>
> current build (statically linked) -
>
> [root@host fio]# ./fio.static --name=test --ioengine=null --size=100g
> --rw=randread --norandommap --gtod_reduce=1
> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=null, iodepth=1
> fio-3.20-70-g9888
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
> test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
> read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
> bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
> stdev=38.59, samples=52
> iops : min=969293, max=1015596, avg=1000951.42,
> stdev=9879.80, samples=52
> cpu : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s
> (4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec
>
> With patches applied[0] -
>
> [root@host fio]# ./fio --name=test --ioengine=null --size=100g
> --rw=randread --norandommap --gtod_reduce=1
> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=null, iodepth=1
> fio-3.20-70-g9888
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
> test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
> read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
> bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
> stdev=45.93, samples=52
> iops : min=960962, max=1021038, avg=1007067.92,
> stdev=11758.92, samples=52
> cpu : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s
> (4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec
>
> I wasn't expecting dynamic linking to have a performance impact and
> the results seem to agree.
That's nice to see. I'd probably add --cpus_allowed=0 to it for
better locality, and then run 10 tests back-to-back with each
just to be sure. There will be some fluctuations, but that should
be enough for me to feel comfortable.
1000K IOPS is pretty slow though, what are you running this on?
Pretty sure my laptop does 10x that.
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 3/3] fio: suggest installing engine package when missing
2020-06-30 19:29 ` Yigal Korman
@ 2020-06-30 19:41 ` Jens Axboe
0 siblings, 0 replies; 16+ messages in thread
From: Jens Axboe @ 2020-06-30 19:41 UTC (permalink / raw)
To: Yigal Korman, Jeff Moyer; +Cc: fio
On 6/30/20 1:29 PM, Yigal Korman wrote:
> On Mon, Jun 29, 2020 at 6:25 PM Jeff Moyer <jmoyer@redhat.com> wrote:
>>
>> Yigal Korman <ykorman@gmail.com> writes:
>>
>>> Signed-off-by: Yigal Korman <ykorman@gmail.com>
>>> ---
>>> ioengines.c | 8 +++++++-
>>> 1 file changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/ioengines.c b/ioengines.c
>>> index 78262bc1..20224e59 100644
>>> --- a/ioengines.c
>>> +++ b/ioengines.c
>>> @@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
>>> static void *dlopen_external(struct thread_data *td, const char *engine)
>>> {
>>> char engine_path[PATH_MAX];
>>> + void *dlhandle;
>>>
>>> sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
>>>
>>> - return dlopen(engine_path, RTLD_LAZY);
>>> + dlhandle = dlopen(engine_path, RTLD_LAZY);
>>> + if (!dlhandle)
>>> + log_info("Engine not available; Install the fio-engine-%s package\n",
>>> + engine);
>>> +
>>> + return dlhandle;
>>> }
>>> #else
>>> #define dlopen_external(td, engine) (NULL)
>>
>> This assumes fio is part of a package, and that's not always (often?)
>> the case. It may be that the engine simply wasn't built.
>
> I agree.
> This patch is more of a showcase of how to help package maintainers
> transition users to the new packaging model.
> I will drop it from the next patchset version.
> This patch might have made more sense if I had implemented a whitelist
> of engines that were compiled dynamically.
I'd say keep it, but make it a bit more generic so it applies to self
built installs too.
"IO Engine %s not available. Either the engine name is invalid, or you need
to Install the fio-engine-%s package, or build fio with the necessary
libraries available."
Something like that.
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-30 19:35 ` Jens Axboe
@ 2020-06-30 19:42 ` Jens Axboe
2020-06-30 20:08 ` Yigal Korman
0 siblings, 1 reply; 16+ messages in thread
From: Jens Axboe @ 2020-06-30 19:42 UTC (permalink / raw)
To: Yigal Korman; +Cc: fio
On 6/30/20 1:35 PM, Jens Axboe wrote:
> On 6/30/20 1:25 PM, Yigal Korman wrote:
>> On Tue, Jun 30, 2020 at 12:07 AM Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>> On 6/28/20 1:27 PM, Yigal Korman wrote:
>>>> When enabled, some of the more dependency-heavy internal engines are
>>>> converted to "plugin" engines, i.e. they are built into separate object
>>>> files and are loaded by fio on demand.
>>>> This helps downstream distros package these engines separately and not
>>>> force a long list of package dependencies from the base fio package.
>>>
>>> How does this impact the performance of the engine? It'd be interesting
>>> to run a test with something ala:
>>>
>>> fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
>>>
>>> with the current build, then apply these patches (and turn the null engine
>>> into an externally loadable engine, of course), and re-run the test case.
>>>
>>> For what it's worth, I like the change in general, as dependencies do
>>> pile on. But I'd like to ensure that we're not taking a performance hit
>>> for something like this.
>>
>> Great to hear.
>>
>> Here are the results of the run you suggested:
>>
>> current build (statically linked) -
>>
>> [root@host fio]# ./fio.static --name=test --ioengine=null --size=100g
>> --rw=randread --norandommap --gtod_reduce=1
>> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>> 4096B-4096B, ioengine=null, iodepth=1
>> fio-3.20-70-g9888
>> Starting 1 process
>> Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
>> test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
>> read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
>> bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
>> stdev=38.59, samples=52
>> iops : min=969293, max=1015596, avg=1000951.42,
>> stdev=9879.80, samples=52
>> cpu : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=1
>>
>> Run status group 0 (all jobs):
>> READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s
>> (4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec
>>
>> With patches applied[0] -
>>
>> [root@host fio]# ./fio --name=test --ioengine=null --size=100g
>> --rw=randread --norandommap --gtod_reduce=1
>> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>> 4096B-4096B, ioengine=null, iodepth=1
>> fio-3.20-70-g9888
>> Starting 1 process
>> Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
>> test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
>> read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
>> bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
>> stdev=45.93, samples=52
>> iops : min=960962, max=1021038, avg=1007067.92,
>> stdev=11758.92, samples=52
>> cpu : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>> latency : target=0, window=0, percentile=100.00%, depth=1
>>
>> Run status group 0 (all jobs):
>> READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s
>> (4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec
>>
>> I wasn't expecting dynamic linking to have a performance impact and
>> the results seem to agree.
>
> That's nice to see. I'd probably add --cpus_allowed=0 to it for
> better locality, and then run 10 tests back-to-back with each
> just to be sure. There will be some fluctuations, but that should
> be enough for me to feel comfortable.
>
> 1000K IOPS is pretty slow though, what are you running this on?
> Pretty sure my laptop does 10x that.
Ran the testing here, and nothing statistically significant.
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-30 19:42 ` Jens Axboe
@ 2020-06-30 20:08 ` Yigal Korman
2020-07-01 1:02 ` Jens Axboe
0 siblings, 1 reply; 16+ messages in thread
From: Yigal Korman @ 2020-06-30 20:08 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
On Tue, Jun 30, 2020 at 10:42 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 6/30/20 1:35 PM, Jens Axboe wrote:
> > On 6/30/20 1:25 PM, Yigal Korman wrote:
> >> On Tue, Jun 30, 2020 at 12:07 AM Jens Axboe <axboe@kernel.dk> wrote:
> >>>
> >>> On 6/28/20 1:27 PM, Yigal Korman wrote:
> >>>> When enabled, some of the more dependency-heavy internal engines are
> >>>> converted to "plugin" engines, i.e. they are built into separate object
> >>>> files and are loaded by fio on demand.
> >>>> This helps downstream distros package these engines separately and not
> >>>> force a long list of package dependencies from the base fio package.
> >>>
> >>> How does this impact the performance of the engine? It'd be interesting
> >>> to run a test with something ala:
> >>>
> >>> fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
> >>>
> >>> with the current build, then apply these patches (and turn the null engine
> >>> into an externally loadable engine, of course), and re-run the test case.
> >>>
> >>> For what it's worth, I like the change in general, as dependencies do
> >>> pile on. But I'd like to ensure that we're not taking a performance hit
> >>> for something like this.
> >>
> >> Great to hear.
> >>
> >> Here are the results of the run you suggested:
> >>
> >> current build (statically linked) -
> >>
> >> [root@host fio]# ./fio.static --name=test --ioengine=null --size=100g
> >> --rw=randread --norandommap --gtod_reduce=1
> >> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> >> 4096B-4096B, ioengine=null, iodepth=1
> >> fio-3.20-70-g9888
> >> Starting 1 process
> >> Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
> >> test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
> >> read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
> >> bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
> >> stdev=38.59, samples=52
> >> iops : min=969293, max=1015596, avg=1000951.42,
> >> stdev=9879.80, samples=52
> >> cpu : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
> >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> >> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> >> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> >> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >> latency : target=0, window=0, percentile=100.00%, depth=1
> >>
> >> Run status group 0 (all jobs):
> >> READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s
> >> (4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec
> >>
> >> With patches applied[0] -
> >>
> >> [root@host fio]# ./fio --name=test --ioengine=null --size=100g
> >> --rw=randread --norandommap --gtod_reduce=1
> >> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> >> 4096B-4096B, ioengine=null, iodepth=1
> >> fio-3.20-70-g9888
> >> Starting 1 process
> >> Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
> >> test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
> >> read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
> >> bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
> >> stdev=45.93, samples=52
> >> iops : min=960962, max=1021038, avg=1007067.92,
> >> stdev=11758.92, samples=52
> >> cpu : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
> >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> >> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> >> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> >> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> >> latency : target=0, window=0, percentile=100.00%, depth=1
> >>
> >> Run status group 0 (all jobs):
> >> READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s
> >> (4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec
> >>
> >> I wasn't expecting dynamic linking to have a performance impact and
> >> the results seem to agree.
> >
> > That's nice to see. I'd probably add --cpus_allowed=0 to it for
> > better locality, and then run 10 tests back-to-back with each
> > just to be sure. There will be some fluctuations, but that should
> > be enough for me to feel comfortable.
> >
> > 1000K IOPS is pretty slow though, what are you running this on?
> > Pretty sure my laptop does 10x that.
Yeah, a tiny VM on my laptop for playing around with the package
manager, not a good candidate for perf testing.
I re-ran as you suggested on the host (i5-6200U @ 2.30GHz), see results here[0].
>
> Ran the testing here, and nothing statistically significant.
Yes, I got the same. Thanks!
Would you like a pull request for the patchset? (after I fix the last patch)
Regards,
Yigal
[0] https://gist.github.com/ykorman/da92200cd79b22f441b98a57ea28726c
>
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v1 2/3] configure: new --dynamic-libengines build option
2020-06-30 20:08 ` Yigal Korman
@ 2020-07-01 1:02 ` Jens Axboe
0 siblings, 0 replies; 16+ messages in thread
From: Jens Axboe @ 2020-07-01 1:02 UTC (permalink / raw)
To: Yigal Korman; +Cc: fio
On 6/30/20 2:08 PM, Yigal Korman wrote:
> On Tue, Jun 30, 2020 at 10:42 PM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 6/30/20 1:35 PM, Jens Axboe wrote:
>>> On 6/30/20 1:25 PM, Yigal Korman wrote:
>>>> On Tue, Jun 30, 2020 at 12:07 AM Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>> On 6/28/20 1:27 PM, Yigal Korman wrote:
>>>>>> When enabled, some of the more dependency-heavy internal engines are
>>>>>> converted to "plugin" engines, i.e. they are built into separate object
>>>>>> files and are loaded by fio on demand.
>>>>>> This helps downstream distros package these engines separately and not
>>>>>> force a long list of package dependencies from the base fio package.
>>>>>
>>>>> How does this impact the performance of the engine? It'd be interesting
>>>>> to run a test with something ala:
>>>>>
>>>>> fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
>>>>>
>>>>> with the current build, then apply these patches (and turn the null engine
>>>>> into an externally loadable engine, of course), and re-run the test case.
>>>>>
>>>>> For what it's worth, I like the change in general, as dependencies do
>>>>> pile on. But I'd like to ensure that we're not taking a performance hit
>>>>> for something like this.
>>>>
>>>> Great to hear.
>>>>
>>>> Here are the results of the run you suggested:
>>>>
>>>> current build (statically linked) -
>>>>
>>>> [root@host fio]# ./fio.static --name=test --ioengine=null --size=100g
>>>> --rw=randread --norandommap --gtod_reduce=1
>>>> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>>>> 4096B-4096B, ioengine=null, iodepth=1
>>>> fio-3.20-70-g9888
>>>> Starting 1 process
>>>> Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
>>>> test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
>>>> read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
>>>> bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
>>>> stdev=38.59, samples=52
>>>> iops : min=969293, max=1015596, avg=1000951.42,
>>>> stdev=9879.80, samples=52
>>>> cpu : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
>>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>>>> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>>> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>>> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>>
>>>> Run status group 0 (all jobs):
>>>> READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s
>>>> (4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec
>>>>
>>>> With patches applied[0] -
>>>>
>>>> [root@host fio]# ./fio --name=test --ioengine=null --size=100g
>>>> --rw=randread --norandommap --gtod_reduce=1
>>>> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>>>> 4096B-4096B, ioengine=null, iodepth=1
>>>> fio-3.20-70-g9888
>>>> Starting 1 process
>>>> Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
>>>> test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
>>>> read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
>>>> bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
>>>> stdev=45.93, samples=52
>>>> iops : min=960962, max=1021038, avg=1007067.92,
>>>> stdev=11758.92, samples=52
>>>> cpu : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
>>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>>>> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>>> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>>> issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>>
>>>> Run status group 0 (all jobs):
>>>> READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s
>>>> (4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec
>>>>
>>>> I wasn't expecting dynamic linking to have a performance impact and
>>>> the results seem to agree.
>>>
>>> That's nice to see. I'd probably add --cpus_allowed=0 to it for
>>> better locality, and then run 10 tests back-to-back with each
>>> just to be sure. There will be some fluctuations, but that should
>>> be enough for me to feel comfortable.
>>>
>>> 1000K IOPS is pretty slow though, what are you running this on?
>>> Pretty sure my laptop does 10x that.
>
> Yeah, a tiny VM on my laptop for playing around with the package
> manager, not a good candidate for perf testing.
> I re-ran as you suggested on the host (i5-6200U @ 2.30GHz), see results here[0].
>
>>
>> Ran the testing here, and nothing statistically significant.
>
> Yes, I got the same. Thanks!
>
> Would you like a pull request for the patchset? (after I fix the last
> patch)
Either that, or just resend the patches, I'm pretty happy with either
approach. GitHub pull requests I do manually anyway, so it's about the
same amount of work. I tend to prefer patches as they are easier to
review, I really don't like viewing patches on GH at all.
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-07-01 1:02 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-28 19:27 [PATCH v1 0/3] fio: option to make engines load dynamically Yigal Korman
2020-06-28 19:27 ` [PATCH v1 1/3] configure/Makefile: engine LIBS consistency Yigal Korman
2020-06-28 19:27 ` [PATCH v1 2/3] configure: new --dynamic-libengines build option Yigal Korman
2020-06-29 21:07 ` Jens Axboe
2020-06-30 19:25 ` Yigal Korman
2020-06-30 19:35 ` Jens Axboe
2020-06-30 19:42 ` Jens Axboe
2020-06-30 20:08 ` Yigal Korman
2020-07-01 1:02 ` Jens Axboe
2020-06-28 19:27 ` [PATCH v1 3/3] fio: suggest installing engine package when missing Yigal Korman
2020-06-29 15:25 ` Jeff Moyer
2020-06-30 19:29 ` Yigal Korman
2020-06-30 19:41 ` Jens Axboe
2020-06-29 15:27 ` [PATCH v1 0/3] fio: option to make engines load dynamically Jeff Moyer
2020-06-29 18:44 ` Yigal Korman
2020-06-29 18:47 ` Yigal Korman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.