[PATCH v2 0/5] (now) five cleanups

[PATCH v2 0/5] (now) five cleanups

[Vincent Fu]

Jens, please consider these five cleanup patches.

Changes from v1 (cleanups, parse /proc/meminfo for huge page size):
- Drop patch that parses /proc/meminfo
- Documentation patch about huge page sizes fixes default size



Vincent Fu (5):
  steadystate: delete incorrect comment
  configure: refer to zlib1g-dev package for zlib support
  HOWTO: add blank line for prettier formatting
  t/run-fio-tests: improve json data decoding
  docs: update discussion of huge page sizes

 HOWTO.rst          | 26 +++++++++++++++-----------
 configure          |  2 +-
 fio.1              | 19 ++++++++++---------
 steadystate.c      |  7 -------
 t/run-fio-tests.py | 20 +++++++-------------
 5 files changed, 33 insertions(+), 41 deletions(-)

-- 
2.25.1

[PATCH v2 2/5] configure: refer to zlib1g-dev package for zlib support

[Vincent Fu]

Recent Debian-based distributions provide zlib support in the zlib1g-dev
package.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
---
 configure | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure b/configure
index 95b60bb7..4ee536a0 100755
--- a/configure
+++ b/configure
@@ -3142,7 +3142,7 @@ if test "$libzbc" = "yes" ; then
   output_sym "CONFIG_LIBZBC"
 fi
 if test "$zlib" = "no" ; then
-  echo "Consider installing zlib-dev (zlib-devel, some fio features depend on it."
+  echo "Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it."
   if test "$build_static" = "yes"; then
     echo "Note that some distros have separate packages for static libraries."
   fi
-- 
2.25.1

[PATCH v2 1/5] steadystate: delete incorrect comment

[Vincent Fu]

Fio actually does not begin collecting steady state data until the
steady state ramp time has expired. Remove the comment that said steady
state data is collected from the start of the job.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
---
 steadystate.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/steadystate.c b/steadystate.c
index 2e3da1db..ad19318c 100644
--- a/steadystate.c
+++ b/steadystate.c
@@ -250,13 +250,6 @@ int steadystate_check(void)
 		rate_time = mtime_since(&ss->prev_time, &now);
 		memcpy(&ss->prev_time, &now, sizeof(now));
 
-		/*
-		 * Begin monitoring when job starts but don't actually use
-		 * data in checking stopping criterion until ss->ramp_time is
-		 * over. This ensures that we will have a sane value in
-		 * prev_iops/bw the first time through after ss->ramp_time
-		 * is done.
-		 */
 		if (ss->state & FIO_SS_RAMP_OVER) {
 			group_bw += 1000 * (td_bytes - ss->prev_bytes) / rate_time;
 			group_iops += 1000 * (td_iops - ss->prev_iops) / rate_time;
-- 
2.25.1

[PATCH v2 3/5] HOWTO: add blank line for prettier formatting

[Vincent Fu]

There needs to be a blank line between the name of an option and its
description in order for the formatting to look nice at
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-ignore-zone-limits,

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
---
 HOWTO.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/HOWTO.rst b/HOWTO.rst
index 84bea5c5..eee386c1 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -1064,6 +1064,7 @@ Target file/device
 	thread/process.
 
 .. option:: ignore_zone_limits=bool
+
 	If this option is used, fio will ignore the maximum number of open
 	zones limit of the zoned block device in use, thus allowing the
 	option :option:`max_open_zones` value to be larger than the device
-- 
2.25.1

[PATCH v2 5/5] docs: update discussion of huge page sizes

[Vincent Fu]

Note that the default huge page size is either 2 or 4 MiB, depending on
the platform. Also mention /sys/kernel/mm/hugepages/ as another place to
see the supported huge page sizes.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
---
 HOWTO.rst | 25 ++++++++++++++-----------
 fio.1     | 19 ++++++++++---------
 2 files changed, 24 insertions(+), 20 deletions(-)

diff --git a/HOWTO.rst b/HOWTO.rst
index eee386c1..58d02fa2 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -1823,13 +1823,14 @@ Buffers and memory
 	**mmaphuge** to work, the system must have free huge pages allocated. This
 	can normally be checked and set by reading/writing
 	:file:`/proc/sys/vm/nr_hugepages` on a Linux system. Fio assumes a huge page
-	is 4MiB in size. So to calculate the number of huge pages you need for a
-	given job file, add up the I/O depth of all jobs (normally one unless
-	:option:`iodepth` is used) and multiply by the maximum bs set. Then divide
-	that number by the huge page size. You can see the size of the huge pages in
-	:file:`/proc/meminfo`. If no huge pages are allocated by having a non-zero
-	number in `nr_hugepages`, using **mmaphuge** or **shmhuge** will fail. Also
-	see :option:`hugepage-size`.
+        is 2 or 4MiB in size depending on the platform. So to calculate the
+        number of huge pages you need for a given job file, add up the I/O
+        depth of all jobs (normally one unless :option:`iodepth` is used) and
+        multiply by the maximum bs set. Then divide that number by the huge
+        page size. You can see the size of the huge pages in
+        :file:`/proc/meminfo`. If no huge pages are allocated by having a
+        non-zero number in `nr_hugepages`, using **mmaphuge** or **shmhuge**
+        will fail. Also see :option:`hugepage-size`.
 
 	**mmaphuge** also needs to have hugetlbfs mounted and the file location
 	should point there. So if it's mounted in :file:`/huge`, you would use
@@ -1848,10 +1849,12 @@ Buffers and memory
 
 .. option:: hugepage-size=int
 
-	Defines the size of a huge page. Must at least be equal to the system
-	setting, see :file:`/proc/meminfo`. Defaults to 4MiB.  Should probably
-	always be a multiple of megabytes, so using ``hugepage-size=Xm`` is the
-	preferred way to set this to avoid setting a non-pow-2 bad value.
+        Defines the size of a huge page. Must at least be equal to the system
+        setting, see :file:`/proc/meminfo` and
+        :file:`/sys/kernel/mm/hugepages/`. Defaults to 2 or 4MiB depending on
+        the platform.  Should probably always be a multiple of megabytes, so
+        using ``hugepage-size=Xm`` is the preferred way to set this to avoid
+        setting a non-pow-2 bad value.
 
 .. option:: lockmem=int
 
diff --git a/fio.1 b/fio.1
index ded7bbfc..5f057574 100644
--- a/fio.1
+++ b/fio.1
@@ -1631,11 +1631,11 @@ multiplied by the I/O depth given. Note that for \fBshmhuge\fR and
 \fBmmaphuge\fR to work, the system must have free huge pages allocated. This
 can normally be checked and set by reading/writing
 `/proc/sys/vm/nr_hugepages' on a Linux system. Fio assumes a huge page
-is 4MiB in size. So to calculate the number of huge pages you need for a
-given job file, add up the I/O depth of all jobs (normally one unless
-\fBiodepth\fR is used) and multiply by the maximum bs set. Then divide
-that number by the huge page size. You can see the size of the huge pages in
-`/proc/meminfo'. If no huge pages are allocated by having a non-zero
+is 2 or 4MiB in size depending on the platform. So to calculate the number of
+huge pages you need for a given job file, add up the I/O depth of all jobs
+(normally one unless \fBiodepth\fR is used) and multiply by the maximum bs set.
+Then divide that number by the huge page size. You can see the size of the huge
+pages in `/proc/meminfo'. If no huge pages are allocated by having a non-zero
 number in `nr_hugepages', using \fBmmaphuge\fR or \fBshmhuge\fR will fail. Also
 see \fBhugepage\-size\fR.
 .P
@@ -1655,10 +1655,11 @@ of subsequent I/O memory buffers is the sum of the \fBiomem_align\fR and
 \fBbs\fR used.
 .TP
 .BI hugepage\-size \fR=\fPint
-Defines the size of a huge page. Must at least be equal to the system
-setting, see `/proc/meminfo'. Defaults to 4MiB. Should probably
-always be a multiple of megabytes, so using `hugepage\-size=Xm' is the
-preferred way to set this to avoid setting a non-pow-2 bad value.
+Defines the size of a huge page. Must at least be equal to the system setting,
+see `/proc/meminfo' and `/sys/kernel/mm/hugepages/'. Defaults to 2 or 4MiB
+depending on the platform. Should probably always be a multiple of megabytes,
+so using `hugepage\-size=Xm' is the preferred way to set this to avoid setting
+a non-pow-2 bad value.
 .TP
 .BI lockmem \fR=\fPint
 Pin the specified amount of memory with \fBmlock\fR\|(2). Can be used to
-- 
2.25.1

[PATCH v2 4/5] t/run-fio-tests: improve json data decoding

[Vincent Fu]

Instead of skipping up to five lines, just skip everything until the
opening curly brace when trying to decode JSON data.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
---
 t/run-fio-tests.py | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index ecceb67e..32cdbc19 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -311,21 +311,15 @@ class FioJobTest(FioExeTest):
         #
         # Sometimes fio informational messages are included at the top of the
         # JSON output, especially under Windows. Try to decode output as JSON
-        # data, lopping off up to the first four lines
+        # data, skipping everything until the first {
         #
         lines = file_data.splitlines()
-        for i in range(5):
-            file_data = '\n'.join(lines[i:])
-            try:
-                self.json_data = json.loads(file_data)
-            except json.JSONDecodeError:
-                continue
-            else:
-                logging.debug("Test %d: skipped %d lines decoding JSON data", self.testnum, i)
-                return
-
-        self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
-        self.passed = False
+        file_data = '\n'.join(lines[lines.index("{"):])
+        try:
+            self.json_data = json.loads(file_data)
+        except json.JSONDecodeError:
+            self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
+            self.passed = False
 
 
 class FioJobTest_t0005(FioJobTest):
-- 
2.25.1

Re: [PATCH v2 0/5] (now) five cleanups

[Jens Axboe]

On Tue, 24 May 2022 14:23:24 +0000, Vincent Fu wrote:
> Jens, please consider these five cleanup patches.
> 
> Changes from v1 (cleanups, parse /proc/meminfo for huge page size):
> - Drop patch that parses /proc/meminfo
> - Documentation patch about huge page sizes fixes default size
> 
> 
> [...]

Applied, thanks!

[1/5] steadystate: delete incorrect comment
      commit: b29472cbdbce66ad9905b58f04ac5ed9e9be0090
[2/5] configure: refer to zlib1g-dev package for zlib support
      commit: 95f791d3ad7d0cf3b024d51713d4861663ba8ce5
[3/5] HOWTO: add blank line for prettier formatting
      commit: a3a6f1058aa41db20794bee1df0bec4019c14d42
[4/5] t/run-fio-tests: improve json data decoding
      commit: db2637ec09786c5610006414f28c2b21a3c3c165
[5/5] docs: update discussion of huge page sizes
      commit: cb8dcafa69ef6fddd22d41c60083cb68bce92daa

Best regards,
-- 
Jens Axboe