Re: [PATCH v4 1/3] btrfs: add read_policy latency

From: Anand Jain <anand.jain@oracle.com>
To: dsterba@suse.cz, linux-btrfs@vger.kernel.org, dsterba@suse.com,
	josef@toxicpanda.com
Subject: Re: [PATCH v4 1/3] btrfs: add read_policy latency
Date: Sat, 30 Jan 2021 09:08:40 +0800	[thread overview]
Message-ID: <9f406301-c16a-72a5-4ff3-d3bda127895e@oracle.com> (raw)
In-Reply-To: <28b7ef3d-b5b9-f4a6-8d6f-1e6fc1103815@oracle.com>

>> 500m is really small data size for such measurement

I reran the read policy tests with some changes in the fio command
options. Mainly to measure IOPS throughput and latency on the filesystem
with latency-policy and pid-policy.

Each of these tests was run for 3 iterations and the best and worst of
those 3 iterations are shown below.

These workloads are performing read-write which is the most commonly
used workload, on a single type of device (which is nvme here) and two
devices are configured for RAID1.

In all these read-write workloads, pid-policy performed ~25% better than
the latency-policy for both throughput and IOPS, and 3% better on the
latency parameter.

I haven't analyzed these read-write workloads on RAID1c3/RAID1c4 yet,
but RAID1 is more common than other types, IMO.

So I think pid-policy should remain as our default read policy.

However as shown before, pid-policy perform worst in the case of special
configs such as volumes with mixed types of devices. For those special
mixed types of devices, latency-policy performs better than pid-policy.
As tested before typically latency-policy provided ~175% better
throughput performance in the case of mixed types of devices (SSD and
nvme).

Feedbacks welcome.

Fio logs below.

IOPS focused readwrite workload:
fio --filename=/btrfs/foo --size=500GB --direct=1 --rw=randrw --bs=4k 
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based 
--group_reporting --name=iops-randomreadwrite --eta-newline=1

pid [latency] device roundrobin ( 00)
   read: IOPS=40.6k, BW=159MiB/s (166MB/s)(18.6GiB/120002msec)

[pid] latency device roundrobin ( 00)
   read: IOPS=50.7k, BW=198MiB/s (208MB/s)(23.2GiB/120001msec)

IOPS is 25% better with pid policy.

Throughput focused readwrite workload:
fio --filename=/btrfs/foo --size=500GB --direct=1 --rw=randrw --bs=64k 
--ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based 
--group_reporting --name=throughput-randomreadwrite --eta-newline=1

pid [latency] device roundrobin ( 00)
   read: IOPS=8525, BW=533MiB/s (559MB/s)(62.4GiB/120003msec)

[pid] latency device roundrobin ( 00)
   read: IOPS=10.7k, BW=670MiB/s (702MB/s)(78.5GiB/120005msec)

Throughput is 25% better with pid policy

Latency focused readwrite workload:
fio --filename=/btrfs/foo --size=500GB --direct=1 --rw=randrw --bs=4k 
--ioengine=libaio --iodepth=1 --runtime=120 --numjobs=4 --time_based 
--group_reporting --name=latency-randomreadwrite --eta-newline=1
pid [latency] device roundrobin ( 00)
   read: IOPS=59.8k, BW=234MiB/s (245MB/s)(27.4GiB/120003msec)
      lat (usec): min=68, max=826930, avg=1917.20, stdev=4210.90

[pid] latency device roundrobin ( 00)
   read: IOPS=61.9k, BW=242MiB/s (253MB/s)(28.3GiB/120001msec)
      lat (usec): min=64, max=751557, avg=1846.07, stdev=4082.97

Latency is 3% better with pid policy.