[PATCH 0/6] OS specific fixes

[PATCH 0/6] OS specific fixes

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>


Tomohiro Kusumi (6):
  cleanup NetBSD/OpenBSD header
  HOWTO: add OpenBSD to direct I/O unsupported platform
  add ifdef __sun__ for Solaris specific code
  filesetup: add non O_DIRECT direct I/O support for initial layout
    setup
  change fio_set_odirect() prototype not to use int fd
  change os_trim() prototype not to use int fd

 HOWTO             |  2 +-
 file.h            |  1 +
 filesetup.c       | 31 +++++++++++++++++++++++++++++++
 fio.1             |  2 +-
 io_u.c            |  2 +-
 ioengines.c       | 22 ++--------------------
 os/os-android.h   |  4 ++--
 os/os-dragonfly.h |  4 ++--
 os/os-freebsd.h   |  4 ++--
 os/os-linux.h     |  4 ++--
 os/os-mac.h       |  4 ++--
 os/os-netbsd.h    |  6 ++----
 os/os-openbsd.h   |  6 ++----
 os/os-solaris.h   |  4 ++--
 14 files changed, 53 insertions(+), 43 deletions(-)

-- 
2.9.5

[PATCH 1/6] cleanup NetBSD/OpenBSD header

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

* NetBSD's <sys/rb.h> was renamed to <sys/rbtree.h> 7 years ago.
* Drop unneeded undefs.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 os/os-netbsd.h  | 6 ++----
 os/os-openbsd.h | 6 ++----
 2 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/os/os-netbsd.h b/os/os-netbsd.h
index eac76cf..682a11c 100644
--- a/os/os-netbsd.h
+++ b/os/os-netbsd.h
@@ -11,9 +11,9 @@
 #include <sys/dkio.h>
 #include <sys/disklabel.h>
 #include <sys/endian.h>
-/* XXX hack to avoid confilcts between rbtree.h and <sys/rb.h> */
-#define	rb_node	_rb_node
 #include <sys/sysctl.h>
+
+/* XXX hack to avoid confilcts between rbtree.h and <sys/rbtree.h> */
 #undef rb_node
 #undef rb_left
 #undef rb_right
@@ -26,8 +26,6 @@
 #define FIO_HAVE_FS_STAT
 #define FIO_HAVE_GETTID
 
-#undef	FIO_HAVE_CPU_AFFINITY	/* doesn't exist */
-
 #define OS_MAP_ANON		MAP_ANON
 
 #ifndef PTHREAD_STACK_MIN
diff --git a/os/os-openbsd.h b/os/os-openbsd.h
index 675bf89..b4c02c9 100644
--- a/os/os-openbsd.h
+++ b/os/os-openbsd.h
@@ -11,23 +11,21 @@
 #include <sys/disklabel.h>
 #include <sys/endian.h>
 #include <sys/utsname.h>
-/* XXX hack to avoid conflicts between rbtree.h and <sys/tree.h> */
 #include <sys/sysctl.h>
+
+/* XXX hack to avoid conflicts between rbtree.h and <sys/tree.h> */
 #undef RB_BLACK
 #undef RB_RED
 #undef RB_ROOT
 
 #include "../file.h"
 
-#undef  FIO_HAVE_ODIRECT
 #define FIO_USE_GENERIC_RAND
 #define FIO_USE_GENERIC_INIT_RANDOM_STATE
 #define FIO_HAVE_FS_STAT
 #define FIO_HAVE_GETTID
 #define FIO_HAVE_SHM_ATTACH_REMOVED
 
-#undef	FIO_HAVE_CPU_AFFINITY	/* doesn't exist */
-
 #define OS_MAP_ANON		MAP_ANON
 
 #ifndef PTHREAD_STACK_MIN
-- 
2.9.5

[PATCH 2/6] HOWTO: add OpenBSD to direct I/O unsupported platform

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

OpenBSD has neither open(O_DIRECT) nor its kernel specific way.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 HOWTO | 2 +-
 fio.1 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/HOWTO b/HOWTO
index f36d4a7..3a720c3 100644
--- a/HOWTO
+++ b/HOWTO
@@ -947,7 +947,7 @@ I/O type
 .. option:: direct=bool
 
 	If value is true, use non-buffered I/O. This is usually O_DIRECT. Note that
-	ZFS on Solaris doesn't support direct I/O.  On Windows the synchronous
+	OpenBSD and ZFS on Solaris don't support direct I/O.  On Windows the synchronous
 	ioengines don't support direct I/O.  Default: false.
 
 .. option:: atomic=bool
diff --git a/fio.1 b/fio.1
index b8b3da2..5b63dfd 100644
--- a/fio.1
+++ b/fio.1
@@ -717,7 +717,7 @@ read. The two zone options can be used to only do I/O on zones of a file.
 .TP
 .BI direct \fR=\fPbool
 If value is true, use non\-buffered I/O. This is usually O_DIRECT. Note that
-ZFS on Solaris doesn't support direct I/O. On Windows the synchronous
+OpenBSD and ZFS on Solaris don't support direct I/O. On Windows the synchronous
 ioengines don't support direct I/O. Default: false.
 .TP
 .BI atomic \fR=\fPbool
-- 
2.9.5

[PATCH 3/6] add ifdef __sun__ for Solaris specific code

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

ENOTTY on direct I/O setup failure via directio(3C) is Solaris (ZFS)
specific (as also mentioned in ZoL commit[1]).

Linux and many others which support O_DIRECT never come here, but
others supporting direct I/O via OS specific way don't necessarily
have the same ENOTTY behavior or ZFS itself.

This commit brings back the original log before 42f1ee68 for others.

[1] https://github.com/zfsonlinux/zfs/commit/a08c76a8ad63c28384ead72b53a3d7ef73f39357

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 ioengines.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ioengines.c b/ioengines.c
index 6e6e3de..6e135af 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -505,12 +505,15 @@ int td_io_open_file(struct thread_data *td, struct fio_file *f)
 
 		if (ret) {
 			td_verror(td, ret, "fio_set_odirect");
+#if defined(__sun__)
 			if (ret == ENOTTY) { /* ENOTTY suggests RAW device or ZFS */
 				log_err("fio: doing directIO to RAW devices or ZFS not supported\n");
 			} else {
 				log_err("fio: the file system does not seem to support direct IO\n");
 			}
-
+#else
+			log_err("fio: the file system does not seem to support direct IO\n");
+#endif
 			goto err;
 		}
 	}
-- 
2.9.5

[PATCH 4/6] filesetup: add non O_DIRECT direct I/O support for initial layout setup

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

8c43ba62('filesetup: align layout buffer') and
6e344dc3('filesetup: keep OS_O_DIRECT flag when pre-allocating file')
only consider OS with open(O_DIRECT).

This commit adds fio_set_directio() since now OS specific direct I/O
initialization is needed by both real ->fd and temporary ->fd while in
extend_file() to equally support the feature.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 file.h      |  1 +
 filesetup.c | 31 +++++++++++++++++++++++++++++++
 ioengines.c | 25 ++-----------------------
 3 files changed, 34 insertions(+), 23 deletions(-)

diff --git a/file.h b/file.h
index 84daa5f..ad8802d 100644
--- a/file.h
+++ b/file.h
@@ -211,5 +211,6 @@ extern void filesetup_mem_free(void);
 extern void fio_file_reset(struct thread_data *, struct fio_file *);
 extern bool fio_files_done(struct thread_data *);
 extern bool exists_and_not_regfile(const char *);
+extern int fio_set_directio(struct thread_data *, struct fio_file *);
 
 #endif
diff --git a/filesetup.c b/filesetup.c
index a6a94ee..4ceaef6 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -196,6 +196,9 @@ static int extend_file(struct thread_data *td, struct fio_file *f)
 		}
 	}
 
+	if (td->o.odirect && !OS_O_DIRECT && fio_set_directio(td, f))
+		goto err;
+
 	left = f->real_file_size;
 	bs = td->o.max_bs[DDIR_WRITE];
 	if (bs > left)
@@ -1852,3 +1855,31 @@ void filesetup_mem_free(void)
 {
 	free_already_allocated();
 }
+
+/*
+ * This function is for platforms which support direct I/O but not O_DIRECT.
+ */
+int fio_set_directio(struct thread_data *td, struct fio_file *f)
+{
+#ifdef FIO_OS_DIRECTIO
+	int ret = fio_set_odirect(f->fd);
+
+	if (ret) {
+		td_verror(td, ret, "fio_set_directio");
+#if defined(__sun__)
+		if (ret == ENOTTY) { /* ENOTTY suggests RAW device or ZFS */
+			log_err("fio: doing directIO to RAW devices or ZFS not supported\n");
+		} else {
+			log_err("fio: the file system does not seem to support direct IO\n");
+		}
+#else
+		log_err("fio: the file system does not seem to support direct IO\n");
+#endif
+		return -1;
+	}
+
+	return 0;
+#else
+	return -1;
+#endif
+}
diff --git a/ioengines.c b/ioengines.c
index 6e135af..919781c 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -495,29 +495,8 @@ int td_io_open_file(struct thread_data *td, struct fio_file *f)
 	}
 #endif
 
-#ifdef FIO_OS_DIRECTIO
-	/*
-	 * Some OS's have a distinct call to mark the file non-buffered,
-	 * instead of using O_DIRECT (Solaris)
-	 */
-	if (td->o.odirect) {
-		int ret = fio_set_odirect(f->fd);
-
-		if (ret) {
-			td_verror(td, ret, "fio_set_odirect");
-#if defined(__sun__)
-			if (ret == ENOTTY) { /* ENOTTY suggests RAW device or ZFS */
-				log_err("fio: doing directIO to RAW devices or ZFS not supported\n");
-			} else {
-				log_err("fio: the file system does not seem to support direct IO\n");
-			}
-#else
-			log_err("fio: the file system does not seem to support direct IO\n");
-#endif
-			goto err;
-		}
-	}
-#endif
+	if (td->o.odirect && !OS_O_DIRECT && fio_set_directio(td, f))
+		goto err;
 
 done:
 	log_file(td, f, FIO_LOG_OPEN_FILE);
-- 
2.9.5

[PATCH 5/6] change fio_set_odirect() prototype not to use int fd

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

This function should take struct fio_file*, since the idea of integer
fd in unix likes isn't necessarily usable on other platforms (Windows).

In fact, other fd related functions under os/ don't take int fd for
portability even if some of them work fine with int fd at the moment.

(OS headers basically (should)have no dependencies on fio functions
and structures, but struct fio_file is the only exception.)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 filesetup.c     | 2 +-
 os/os-mac.h     | 4 ++--
 os/os-solaris.h | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/filesetup.c b/filesetup.c
index 4ceaef6..c4240d2 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -1862,7 +1862,7 @@ void filesetup_mem_free(void)
 int fio_set_directio(struct thread_data *td, struct fio_file *f)
 {
 #ifdef FIO_OS_DIRECTIO
-	int ret = fio_set_odirect(f->fd);
+	int ret = fio_set_odirect(f);
 
 	if (ret) {
 		td_verror(td, ret, "fio_set_directio");
diff --git a/os/os-mac.h b/os/os-mac.h
index a1536c7..92a60ee 100644
--- a/os/os-mac.h
+++ b/os/os-mac.h
@@ -41,9 +41,9 @@ typedef unsigned int clockid_t;
 #endif
 
 #define FIO_OS_DIRECTIO
-static inline int fio_set_odirect(int fd)
+static inline int fio_set_odirect(struct fio_file *f)
 {
-	if (fcntl(fd, F_NOCACHE, 1) == -1)
+	if (fcntl(f->fd, F_NOCACHE, 1) == -1)
 		return errno;
 	return 0;
 }
diff --git a/os/os-solaris.h b/os/os-solaris.h
index 8f8f53b..6af25d2 100644
--- a/os/os-solaris.h
+++ b/os/os-solaris.h
@@ -85,9 +85,9 @@ static inline long os_random_long(os_random_state_t *rs)
 
 #define FIO_OS_DIRECTIO
 extern int directio(int, int);
-static inline int fio_set_odirect(int fd)
+static inline int fio_set_odirect(struct fio_file *f)
 {
-	if (directio(fd, DIRECTIO_ON) < 0)
+	if (directio(f->fd, DIRECTIO_ON) < 0)
 		return errno;
 
 	return 0;
-- 
2.9.5

[PATCH 6/6] change os_trim() prototype not to use int fd

[kusumi.tomohiro]

From: Tomohiro Kusumi <tkusumi@tuxera.com>

This is the same as the previous commit. The offset and length are
universal regardless of platforms, but int fd is not, thus it should
take struct fio_file*.

In fact, other fd related functions under os/ don't take int fd for
portability even if some of them work fine with int fd at the moment.

(OS headers basically (should)have no dependencies on fio functions
and structures, but struct fio_file is the only exception.)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
---
 io_u.c            | 2 +-
 os/os-android.h   | 4 ++--
 os/os-dragonfly.h | 4 ++--
 os/os-freebsd.h   | 4 ++--
 os/os-linux.h     | 4 ++--
 5 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/io_u.c b/io_u.c
index ed8e84a..db043e4 100644
--- a/io_u.c
+++ b/io_u.c
@@ -2188,7 +2188,7 @@ int do_io_u_trim(const struct thread_data *td, struct io_u *io_u)
 	struct fio_file *f = io_u->file;
 	int ret;
 
-	ret = os_trim(f->fd, io_u->offset, io_u->xfer_buflen);
+	ret = os_trim(f, io_u->offset, io_u->xfer_buflen);
 	if (!ret)
 		return io_u->xfer_buflen;
 
diff --git a/os/os-android.h b/os/os-android.h
index b217daa..bb590e4 100644
--- a/os/os-android.h
+++ b/os/os-android.h
@@ -274,7 +274,7 @@ static inline unsigned long long get_fs_free_size(const char *path)
 	return ret;
 }
 
-static inline int os_trim(int fd, unsigned long long start,
+static inline int os_trim(struct fio_file *f, unsigned long long start,
 			  unsigned long long len)
 {
 	uint64_t range[2];
@@ -282,7 +282,7 @@ static inline int os_trim(int fd, unsigned long long start,
 	range[0] = start;
 	range[1] = len;
 
-	if (!ioctl(fd, BLKDISCARD, range))
+	if (!ioctl(f->fd, BLKDISCARD, range))
 		return 0;
 
 	return errno;
diff --git a/os/os-dragonfly.h b/os/os-dragonfly.h
index 8d15833..423b236 100644
--- a/os/os-dragonfly.h
+++ b/os/os-dragonfly.h
@@ -216,7 +216,7 @@ static inline unsigned long long get_fs_free_size(const char *path)
 	return ret;
 }
 
-static inline int os_trim(int fd, unsigned long long start,
+static inline int os_trim(struct fio_file *f, unsigned long long start,
 			  unsigned long long len)
 {
 	off_t range[2];
@@ -224,7 +224,7 @@ static inline int os_trim(int fd, unsigned long long start,
 	range[0] = start;
 	range[1] = len;
 
-	if (!ioctl(fd, IOCTLTRIM, range))
+	if (!ioctl(f->fd, IOCTLTRIM, range))
 		return 0;
 
 	return errno;
diff --git a/os/os-freebsd.h b/os/os-freebsd.h
index e6da286..4a7cdeb 100644
--- a/os/os-freebsd.h
+++ b/os/os-freebsd.h
@@ -117,7 +117,7 @@ static inline unsigned long long get_fs_free_size(const char *path)
 	return ret;
 }
 
-static inline int os_trim(int fd, unsigned long long start,
+static inline int os_trim(struct fio_file *f, unsigned long long start,
 			  unsigned long long len)
 {
 	off_t range[2];
@@ -125,7 +125,7 @@ static inline int os_trim(int fd, unsigned long long start,
 	range[0] = start;
 	range[1] = len;
 
-	if (!ioctl(fd, DIOCGDELETE, range))
+	if (!ioctl(f->fd, DIOCGDELETE, range))
 		return 0;
 
 	return errno;
diff --git a/os/os-linux.h b/os/os-linux.h
index e7d600d..1ad6ebd 100644
--- a/os/os-linux.h
+++ b/os/os-linux.h
@@ -281,7 +281,7 @@ static inline unsigned long long get_fs_free_size(const char *path)
 	return ret;
 }
 
-static inline int os_trim(int fd, unsigned long long start,
+static inline int os_trim(struct fio_file *f, unsigned long long start,
 			  unsigned long long len)
 {
 	uint64_t range[2];
@@ -289,7 +289,7 @@ static inline int os_trim(int fd, unsigned long long start,
 	range[0] = start;
 	range[1] = len;
 
-	if (!ioctl(fd, BLKDISCARD, range))
+	if (!ioctl(f->fd, BLKDISCARD, range))
 		return 0;
 
 	return errno;
-- 
2.9.5

Re: [PATCH 0/6] OS specific fixes

[Jens Axboe]

On Wed, Aug 30 2017, kusumi.tomohiro@gmail.com wrote:
> From: Tomohiro Kusumi <tkusumi@tuxera.com>
> 
> 
> Tomohiro Kusumi (6):
>   cleanup NetBSD/OpenBSD header
>   HOWTO: add OpenBSD to direct I/O unsupported platform
>   add ifdef __sun__ for Solaris specific code
>   filesetup: add non O_DIRECT direct I/O support for initial layout
>     setup
>   change fio_set_odirect() prototype not to use int fd
>   change os_trim() prototype not to use int fd

Looks good, applied.

-- 
Jens Axboe