* [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-11 9:41 [PATCH v3 0/4] btrfs: read_policy types latency, device and round-robin Anand Jain
@ 2021-01-11 9:41 ` Anand Jain
2021-01-19 19:36 ` Josef Bacik
2021-01-20 10:27 ` Michal Rostecki
2021-01-11 9:41 ` [PATCH v3 2/4] btrfs: introduce new device-state read_preferred Anand Jain
` (2 subsequent siblings)
3 siblings, 2 replies; 15+ messages in thread
From: Anand Jain @ 2021-01-11 9:41 UTC (permalink / raw)
To: linux-btrfs; +Cc: dsterba, josef, Anand Jain
The read policy type latency routes the read IO based on the historical
average wait-time experienced by the read IOs through the individual
device. This patch obtains the historical read IO stats from the kernel
block layer and calculates its average.
Example usage:
echo "latency" > /sys/fs/btrfs/$uuid/read_policy
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v3: The block layer commit 0d02129e76ed (block: merge struct block_device and
struct hd_struct) has changed the first argument in the function
part_stat_read_all() in 5.11-rc1. So the compilation will fail. This patch
fixes it.
Commit log updated.
v2: Use btrfs_debug_rl() instead of btrfs_info_rl()
It is better we have this debug until we test this on at least few
hardwares.
Drop the unrelated changes.
Update change log.
v1: Drop part_stat_read_all instead use part_stat_read
Drop inflight
fs/btrfs/sysfs.c | 3 ++-
fs/btrfs/volumes.c | 38 ++++++++++++++++++++++++++++++++++++++
fs/btrfs/volumes.h | 2 ++
3 files changed, 42 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 19b9fffa2c9c..96ca7bef6357 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -915,7 +915,8 @@ static bool strmatch(const char *buffer, const char *string)
return false;
}
-static const char * const btrfs_read_policy_name[] = { "pid" };
+/* Must follow the order as in enum btrfs_read_policy */
+static const char * const btrfs_read_policy_name[] = { "pid", "latency" };
static ssize_t btrfs_read_policy_show(struct kobject *kobj,
struct kobj_attribute *a, char *buf)
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index f4037a6bd926..f7a0a83d2cd4 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -14,6 +14,7 @@
#include <linux/semaphore.h>
#include <linux/uuid.h>
#include <linux/list_sort.h>
+#include <linux/part_stat.h>
#include "misc.h"
#include "ctree.h"
#include "extent_map.h"
@@ -5490,6 +5491,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
return ret;
}
+static int btrfs_find_best_stripe(struct btrfs_fs_info *fs_info,
+ struct map_lookup *map, int first,
+ int num_stripe)
+{
+ u64 est_wait = 0;
+ int best_stripe = 0;
+ int index;
+
+ for (index = first; index < first + num_stripe; index++) {
+ u64 read_wait;
+ u64 avg_wait = 0;
+ unsigned long read_ios;
+ struct btrfs_device *device = map->stripes[index].dev;
+
+ read_wait = part_stat_read(device->bdev, nsecs[READ]);
+ read_ios = part_stat_read(device->bdev, ios[READ]);
+
+ if (read_wait && read_ios && read_wait >= read_ios)
+ avg_wait = div_u64(read_wait, read_ios);
+ else
+ btrfs_debug_rl(device->fs_devices->fs_info,
+ "devid: %llu avg_wait ZERO read_wait %llu read_ios %lu",
+ device->devid, read_wait, read_ios);
+
+ if (est_wait == 0 || est_wait > avg_wait) {
+ est_wait = avg_wait;
+ best_stripe = index;
+ }
+ }
+
+ return best_stripe;
+}
+
static int find_live_mirror(struct btrfs_fs_info *fs_info,
struct map_lookup *map, int first,
int dev_replace_is_ongoing)
@@ -5519,6 +5553,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
case BTRFS_READ_POLICY_PID:
preferred_mirror = first + (current->pid % num_stripes);
break;
+ case BTRFS_READ_POLICY_LATENCY:
+ preferred_mirror = btrfs_find_best_stripe(fs_info, map, first,
+ num_stripes);
+ break;
}
if (dev_replace_is_ongoing &&
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 1997a4649a66..71ba1f0e93f4 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -222,6 +222,8 @@ enum btrfs_chunk_allocation_policy {
enum btrfs_read_policy {
/* Use process PID to choose the stripe */
BTRFS_READ_POLICY_PID,
+ /* Find and use device with the lowest latency */
+ BTRFS_READ_POLICY_LATENCY,
BTRFS_NR_READ_POLICY,
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-11 9:41 ` [PATCH v3 1/4] btrfs: add read_policy latency Anand Jain
@ 2021-01-19 19:36 ` Josef Bacik
2021-01-20 2:43 ` Anand Jain
2021-01-20 10:27 ` Michal Rostecki
1 sibling, 1 reply; 15+ messages in thread
From: Josef Bacik @ 2021-01-19 19:36 UTC (permalink / raw)
To: Anand Jain, linux-btrfs; +Cc: dsterba
On 1/11/21 4:41 AM, Anand Jain wrote:
> The read policy type latency routes the read IO based on the historical
> average wait-time experienced by the read IOs through the individual
> device. This patch obtains the historical read IO stats from the kernel
> block layer and calculates its average.
>
> Example usage:
> echo "latency" > /sys/fs/btrfs/$uuid/read_policy
>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> ---
> v3: The block layer commit 0d02129e76ed (block: merge struct block_device and
> struct hd_struct) has changed the first argument in the function
> part_stat_read_all() in 5.11-rc1. So the compilation will fail. This patch
> fixes it.
> Commit log updated.
>
> v2: Use btrfs_debug_rl() instead of btrfs_info_rl()
> It is better we have this debug until we test this on at least few
> hardwares.
> Drop the unrelated changes.
> Update change log.
>
> v1: Drop part_stat_read_all instead use part_stat_read
> Drop inflight
>
>
> fs/btrfs/sysfs.c | 3 ++-
> fs/btrfs/volumes.c | 38 ++++++++++++++++++++++++++++++++++++++
> fs/btrfs/volumes.h | 2 ++
> 3 files changed, 42 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> index 19b9fffa2c9c..96ca7bef6357 100644
> --- a/fs/btrfs/sysfs.c
> +++ b/fs/btrfs/sysfs.c
> @@ -915,7 +915,8 @@ static bool strmatch(const char *buffer, const char *string)
> return false;
> }
>
> -static const char * const btrfs_read_policy_name[] = { "pid" };
> +/* Must follow the order as in enum btrfs_read_policy */
> +static const char * const btrfs_read_policy_name[] = { "pid", "latency" };
>
> static ssize_t btrfs_read_policy_show(struct kobject *kobj,
> struct kobj_attribute *a, char *buf)
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index f4037a6bd926..f7a0a83d2cd4 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -14,6 +14,7 @@
> #include <linux/semaphore.h>
> #include <linux/uuid.h>
> #include <linux/list_sort.h>
> +#include <linux/part_stat.h>
> #include "misc.h"
> #include "ctree.h"
> #include "extent_map.h"
> @@ -5490,6 +5491,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
> return ret;
> }
>
> +static int btrfs_find_best_stripe(struct btrfs_fs_info *fs_info,
> + struct map_lookup *map, int first,
> + int num_stripe)
> +{
> + u64 est_wait = 0;
> + int best_stripe = 0;
> + int index;
> +
> + for (index = first; index < first + num_stripe; index++) {
> + u64 read_wait;
> + u64 avg_wait = 0;
> + unsigned long read_ios;
> + struct btrfs_device *device = map->stripes[index].dev;
> +
> + read_wait = part_stat_read(device->bdev, nsecs[READ]);
> + read_ios = part_stat_read(device->bdev, ios[READ]);
> +
> + if (read_wait && read_ios && read_wait >= read_ios)
> + avg_wait = div_u64(read_wait, read_ios);
> + else
> + btrfs_debug_rl(device->fs_devices->fs_info,
You can just use fs_info here, you already have it. I'm not in love with doing
this check every time, I'd rather cache the results somewhere. However if we're
read-only I can't think of a mechanism we could piggy back on, RW we could just
do it every transaction commit. Fix the fs_info thing and you can add
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Thanks,
Josef
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-19 19:36 ` Josef Bacik
@ 2021-01-20 2:43 ` Anand Jain
0 siblings, 0 replies; 15+ messages in thread
From: Anand Jain @ 2021-01-20 2:43 UTC (permalink / raw)
To: Josef Bacik, linux-btrfs; +Cc: dsterba
On 20/1/21 3:36 am, Josef Bacik wrote:
> On 1/11/21 4:41 AM, Anand Jain wrote:
>> The read policy type latency routes the read IO based on the historical
>> average wait-time experienced by the read IOs through the individual
>> device. This patch obtains the historical read IO stats from the kernel
>> block layer and calculates its average.
>>
>> Example usage:
>> echo "latency" > /sys/fs/btrfs/$uuid/read_policy
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> ---
>> v3: The block layer commit 0d02129e76ed (block: merge struct
>> block_device and
>> struct hd_struct) has changed the first argument in the function
>> part_stat_read_all() in 5.11-rc1. So the compilation will fail.
>> This patch
>> fixes it.
>> Commit log updated.
>>
>> v2: Use btrfs_debug_rl() instead of btrfs_info_rl()
>> It is better we have this debug until we test this on at least few
>> hardwares.
>> Drop the unrelated changes.
>> Update change log.
>>
>> v1: Drop part_stat_read_all instead use part_stat_read
>> Drop inflight
>>
>>
>> fs/btrfs/sysfs.c | 3 ++-
>> fs/btrfs/volumes.c | 38 ++++++++++++++++++++++++++++++++++++++
>> fs/btrfs/volumes.h | 2 ++
>> 3 files changed, 42 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
>> index 19b9fffa2c9c..96ca7bef6357 100644
>> --- a/fs/btrfs/sysfs.c
>> +++ b/fs/btrfs/sysfs.c
>> @@ -915,7 +915,8 @@ static bool strmatch(const char *buffer, const
>> char *string)
>> return false;
>> }
>> -static const char * const btrfs_read_policy_name[] = { "pid" };
>> +/* Must follow the order as in enum btrfs_read_policy */
>> +static const char * const btrfs_read_policy_name[] = { "pid",
>> "latency" };
>> static ssize_t btrfs_read_policy_show(struct kobject *kobj,
>> struct kobj_attribute *a, char *buf)
>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
>> index f4037a6bd926..f7a0a83d2cd4 100644
>> --- a/fs/btrfs/volumes.c
>> +++ b/fs/btrfs/volumes.c
>> @@ -14,6 +14,7 @@
>> #include <linux/semaphore.h>
>> #include <linux/uuid.h>
>> #include <linux/list_sort.h>
>> +#include <linux/part_stat.h>
>> #include "misc.h"
>> #include "ctree.h"
>> #include "extent_map.h"
>> @@ -5490,6 +5491,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info
>> *fs_info, u64 logical, u64 len)
>> return ret;
>> }
>> +static int btrfs_find_best_stripe(struct btrfs_fs_info *fs_info,
>> + struct map_lookup *map, int first,
>> + int num_stripe)
>> +{
>> + u64 est_wait = 0;
>> + int best_stripe = 0;
>> + int index;
>> +
>> + for (index = first; index < first + num_stripe; index++) {
>> + u64 read_wait;
>> + u64 avg_wait = 0;
>> + unsigned long read_ios;
>> + struct btrfs_device *device = map->stripes[index].dev;
>> +
>> + read_wait = part_stat_read(device->bdev, nsecs[READ]);
>> + read_ios = part_stat_read(device->bdev, ios[READ]);
>> +
>> + if (read_wait && read_ios && read_wait >= read_ios)
>> + avg_wait = div_u64(read_wait, read_ios);
>> + else
>> + btrfs_debug_rl(device->fs_devices->fs_info,
>
> You can just use fs_info here, you already have it. I'm not in love
> with doing this check every time, I'd rather cache the results
> somewhere. However if we're read-only I can't think of a mechanism we
> could piggy back on, RW we could just do it every transaction commit.
> Fix the fs_info thing and you can add
>
Oh. I will fix it. Thanks for the review here and in other patches.
-Anand
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
>
> Thanks,
>
> Josef
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-11 9:41 ` [PATCH v3 1/4] btrfs: add read_policy latency Anand Jain
2021-01-19 19:36 ` Josef Bacik
@ 2021-01-20 10:27 ` Michal Rostecki
2021-01-20 12:30 ` Anand Jain
1 sibling, 1 reply; 15+ messages in thread
From: Michal Rostecki @ 2021-01-20 10:27 UTC (permalink / raw)
To: Anand Jain; +Cc: linux-btrfs, dsterba, josef
On Mon, Jan 11, 2021 at 05:41:34PM +0800, Anand Jain wrote:
> The read policy type latency routes the read IO based on the historical
> average wait-time experienced by the read IOs through the individual
> device. This patch obtains the historical read IO stats from the kernel
> block layer and calculates its average.
>
> Example usage:
> echo "latency" > /sys/fs/btrfs/$uuid/read_policy
>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> ---
> v3: The block layer commit 0d02129e76ed (block: merge struct block_device and
> struct hd_struct) has changed the first argument in the function
> part_stat_read_all() in 5.11-rc1. So the compilation will fail. This patch
> fixes it.
> Commit log updated.
>
> v2: Use btrfs_debug_rl() instead of btrfs_info_rl()
> It is better we have this debug until we test this on at least few
> hardwares.
> Drop the unrelated changes.
> Update change log.
>
> v1: Drop part_stat_read_all instead use part_stat_read
> Drop inflight
>
Hi Anand,
I tested this policy with fio and dstat. It performs overall really
well. On my raid1c3 array with two HDDs and one SSD (which is the last
device), I'm getting the following results.
With direct=0:
Run status group 0 (all jobs):
READ: bw=3560MiB/s (3733MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
io=3129GiB (3360GB), run=900003-900013msec
With direct=1:
Run status group 0 (all jobs):
READ: bw=520MiB/s (545MB/s), 64.9MiB/s-65.0MiB/s (68.1MB/s-68.2MB/s),
io=457GiB (490GB), run=900001-900001msec
However, I was also running dstat at the same time and I noticed that
the read stop sometimes for ~15-20 seconds. For example:
----system---- --dsk/sdb-- --dsk/sdc-- --dsk/sdd--
20-01 00:37:21| 0 0 : 0 0 : 509M 0
20-01 00:37:22| 0 0 : 0 0 : 517M 0
20-01 00:37:23| 0 0 : 0 0 : 507M 0
20-01 00:37:24| 0 0 : 0 0 : 518M 0
20-01 00:37:25| 0 0 : 0 0 : 22M 0
20-01 00:37:26| 0 0 : 0 0 : 0 0
20-01 00:37:27| 0 0 : 0 0 : 0 0
20-01 00:37:28| 0 0 : 0 0 : 0 0
20-01 00:37:29| 0 0 : 0 0 : 0 0
20-01 00:37:30| 0 0 : 0 0 : 0 0
20-01 00:37:31| 0 0 : 0 0 : 0 0
20-01 00:37:32| 0 0 : 0 0 : 0 0
20-01 00:37:33| 0 0 : 0 0 : 0 0
20-01 00:37:34| 0 0 : 0 0 : 0 0
20-01 00:37:35| 0 0 : 0 0 : 0 0
20-01 00:37:36| 0 0 : 0 0 : 0 0
20-01 00:37:37| 0 0 : 0 0 : 0 0
20-01 00:37:38| 0 0 : 0 0 : 0 0
20-01 00:37:39| 0 0 : 0 0 : 0 0
20-01 00:37:40| 0 0 : 0 0 : 0 0
20-01 00:37:41| 0 0 : 0 0 : 0 0
20-01 00:37:42| 0 0 : 0 0 : 0 0
20-01 00:37:43| 0 0 : 0 0 : 0 0
20-01 00:37:44| 0 0 : 0 0 : 0 0
20-01 00:37:45| 0 0 : 0 0 : 0 0
20-01 00:37:46| 0 0 : 0 0 : 55M 0
20-01 00:37:47| 0 0 : 0 0 : 516M 0
20-01 00:37:48| 0 0 : 0 0 : 515M 0
20-01 00:37:49| 0 0 : 0 0 : 516M 0
20-01 00:37:50| 0 0 : 0 0 : 520M 0
20-01 00:37:51| 0 0 : 0 0 : 520M 0
20-01 00:37:52| 0 0 : 0 0 : 514M 0
Here is the full log:
https://susepaste.org/16928336
I never noticed that happening with the PID policy. Is that maybe
because of reading the part stats for all CPUs while selecting the
mirror?
Michal
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-20 10:27 ` Michal Rostecki
@ 2021-01-20 12:30 ` Anand Jain
2021-01-20 13:54 ` Michal Rostecki
0 siblings, 1 reply; 15+ messages in thread
From: Anand Jain @ 2021-01-20 12:30 UTC (permalink / raw)
To: Michal Rostecki; +Cc: linux-btrfs, dsterba, josef
> Hi Anand,
>
> I tested this policy with fio and dstat. It performs overall really
> well. On my raid1c3 array with two HDDs and one SSD (which is the last
> device), I'm getting the following results.
>
Michal,
Thank you for verifying. More below...
> With direct=0:
>
> Run status group 0 (all jobs):
> READ: bw=3560MiB/s (3733MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
> io=3129GiB (3360GB), run=900003-900013msec
>
> With direct=1:
>
> Run status group 0 (all jobs):
> READ: bw=520MiB/s (545MB/s), 64.9MiB/s-65.0MiB/s (68.1MB/s-68.2MB/s),
> io=457GiB (490GB), run=900001-900001msec
>
> However, I was also running dstat at the same time and I noticed that
> the read stop sometimes for ~15-20 seconds. For example:
>
> ----system---- --dsk/sdb-- --dsk/sdc-- --dsk/sdd--
> 20-01 00:37:21| 0 0 : 0 0 : 509M 0
> 20-01 00:37:22| 0 0 : 0 0 : 517M 0
> 20-01 00:37:23| 0 0 : 0 0 : 507M 0
> 20-01 00:37:24| 0 0 : 0 0 : 518M 0
> 20-01 00:37:25| 0 0 : 0 0 : 22M 0
> 20-01 00:37:26| 0 0 : 0 0 : 0 0
> 20-01 00:37:27| 0 0 : 0 0 : 0 0
> 20-01 00:37:28| 0 0 : 0 0 : 0 0
> 20-01 00:37:29| 0 0 : 0 0 : 0 0
> 20-01 00:37:30| 0 0 : 0 0 : 0 0
> 20-01 00:37:31| 0 0 : 0 0 : 0 0
> 20-01 00:37:32| 0 0 : 0 0 : 0 0
> 20-01 00:37:33| 0 0 : 0 0 : 0 0
> 20-01 00:37:34| 0 0 : 0 0 : 0 0
> 20-01 00:37:35| 0 0 : 0 0 : 0 0
> 20-01 00:37:36| 0 0 : 0 0 : 0 0
> 20-01 00:37:37| 0 0 : 0 0 : 0 0
> 20-01 00:37:38| 0 0 : 0 0 : 0 0
> 20-01 00:37:39| 0 0 : 0 0 : 0 0
> 20-01 00:37:40| 0 0 : 0 0 : 0 0
> 20-01 00:37:41| 0 0 : 0 0 : 0 0
> 20-01 00:37:42| 0 0 : 0 0 : 0 0
> 20-01 00:37:43| 0 0 : 0 0 : 0 0
> 20-01 00:37:44| 0 0 : 0 0 : 0 0
> 20-01 00:37:45| 0 0 : 0 0 : 0 0
> 20-01 00:37:46| 0 0 : 0 0 : 55M 0
> 20-01 00:37:47| 0 0 : 0 0 : 516M 0
> 20-01 00:37:48| 0 0 : 0 0 : 515M 0
> 20-01 00:37:49| 0 0 : 0 0 : 516M 0
> 20-01 00:37:50| 0 0 : 0 0 : 520M 0
> 20-01 00:37:51| 0 0 : 0 0 : 520M 0
> 20-01 00:37:52| 0 0 : 0 0 : 514M 0
>
> Here is the full log:
>
> https://susepaste.org/16928336
>
> I never noticed that happening with the PID policy. Is that maybe
> because of reading the part stats for all CPUs while selecting the
> mirror?
>
I ran fio tests again, now with dstat in an another window. I don't
notice any such stalls, the read numbers went continuous until fio
finished. Could you please check with the below fio command, also
could you please share your fio command options.
fio \
--filename=/btrfs/largefile \
--directory=/btrfs \
--filesize=50G \
--size=50G \
--bs=64k \
--ioengine=libaio \
--rw=read \
--direct=1 \
--numjobs=1 \
--group_reporting \
--thread \
--name iops-test-job
It is system specific?
Thanks.
Anand
> Michal
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-20 12:30 ` Anand Jain
@ 2021-01-20 13:54 ` Michal Rostecki
2021-01-21 10:45 ` Anand Jain
0 siblings, 1 reply; 15+ messages in thread
From: Michal Rostecki @ 2021-01-20 13:54 UTC (permalink / raw)
To: Anand Jain; +Cc: linux-btrfs, dsterba, josef
On Wed, Jan 20, 2021 at 08:30:56PM +0800, Anand Jain wrote:
> I ran fio tests again, now with dstat in an another window. I don't
> notice any such stalls, the read numbers went continuous until fio
> finished. Could you please check with the below fio command, also
> could you please share your fio command options.
That's the fio config I used:
https://gitlab.com/vadorovsky/playground/-/blob/master/fio/btrfs-raid1-seqread.fio
The main differences seem to be:
- the number of jobs (I used the number of CPU threads)
- direct vs buffered I/O
>
> fio \
> --filename=/btrfs/largefile \
> --directory=/btrfs \
> --filesize=50G \
> --size=50G \
> --bs=64k \
> --ioengine=libaio \
> --rw=read \
> --direct=1 \
> --numjobs=1 \
> --group_reporting \
> --thread \
> --name iops-test-job
>
> It is system specific?
With this command, dstat output looks good:
https://paste.opensuse.org/view/simple/93159623
So I think it might be specific to whether we test direct of buffered
I/O. Or to the number of jobs (single vs multiple jobs). Since the most
of I/O on production environments is usually buffered, I think we should
test with direct=0 too.
Cheers,
Michal
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/4] btrfs: add read_policy latency
2021-01-20 13:54 ` Michal Rostecki
@ 2021-01-21 10:45 ` Anand Jain
0 siblings, 0 replies; 15+ messages in thread
From: Anand Jain @ 2021-01-21 10:45 UTC (permalink / raw)
To: Michal Rostecki; +Cc: linux-btrfs, dsterba, josef
On 20/1/21 9:54 pm, Michal Rostecki wrote:
> On Wed, Jan 20, 2021 at 08:30:56PM +0800, Anand Jain wrote:
>> I ran fio tests again, now with dstat in an another window. I don't
>> notice any such stalls, the read numbers went continuous until fio
>> finished. Could you please check with the below fio command, also
>> could you please share your fio command options.
>
> That's the fio config I used:
>
> https://gitlab.com/vadorovsky/playground/-/blob/master/fio/btrfs-raid1-seqread.fio
>
> The main differences seem to be:
> - the number of jobs (I used the number of CPU threads)
> - direct vs buffered I/O
>
>>
>> fio \
>> --filename=/btrfs/largefile \
>> --directory=/btrfs \
>> --filesize=50G \
>> --size=50G \
>> --bs=64k \
>> --ioengine=libaio \
>> --rw=read \
>> --direct=1 \
>> --numjobs=1 \
>> --group_reporting \
>> --thread \
>> --name iops-test-job
>>
>> It is system specific?
>
> With this command, dstat output looks good:
>
> https://paste.opensuse.org/view/simple/93159623
>
> So I think it might be specific to whether we test direct of buffered
> I/O. Or to the number of jobs (single vs multiple jobs). Since the most
> of I/O on production environments is usually buffered, I think we should
> test with direct=0 too.
>
It explains the stall for now, so it might be reading from the cache
so there was actually no IO to the device.
Agreed it must be tested with the most common type of IO. But as the
cache comes into play I was a bit reluctant.
Thanks, Anand
> Cheers,
> Michal
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 2/4] btrfs: introduce new device-state read_preferred
2021-01-11 9:41 [PATCH v3 0/4] btrfs: read_policy types latency, device and round-robin Anand Jain
2021-01-11 9:41 ` [PATCH v3 1/4] btrfs: add read_policy latency Anand Jain
@ 2021-01-11 9:41 ` Anand Jain
2021-01-19 19:44 ` Josef Bacik
2021-01-11 9:41 ` [PATCH v3 3/4] btrfs: introduce new read_policy device Anand Jain
2021-01-11 9:41 ` [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin Anand Jain
3 siblings, 1 reply; 15+ messages in thread
From: Anand Jain @ 2021-01-11 9:41 UTC (permalink / raw)
To: linux-btrfs; +Cc: dsterba, josef, Anand Jain
This is a preparatory patch and introduces a new device flag
'read_preferred', RW-able using sysfs interface.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v2: C style fixes. Drop space in between '! test_bit' and extra lines
after it.
fs/btrfs/sysfs.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/volumes.h | 1 +
2 files changed, 54 insertions(+)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 96ca7bef6357..b5d6499fbad0 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -1417,11 +1417,64 @@ static ssize_t btrfs_devinfo_writeable_show(struct kobject *kobj,
}
BTRFS_ATTR(devid, writeable, btrfs_devinfo_writeable_show);
+static ssize_t btrfs_devinfo_read_pref_show(struct kobject *kobj,
+ struct kobj_attribute *a, char *buf)
+{
+ int val;
+ struct btrfs_device *device = container_of(kobj, struct btrfs_device,
+ devid_kobj);
+
+ val = !!test_bit(BTRFS_DEV_STATE_READ_PREFERRED, &device->dev_state);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t btrfs_devinfo_read_pref_store(struct kobject *kobj,
+ struct kobj_attribute *a,
+ const char *buf, size_t len)
+{
+ int ret;
+ unsigned long val;
+ struct btrfs_device *device;
+
+ ret = kstrtoul(skip_spaces(buf), 0, &val);
+ if (ret)
+ return ret;
+
+ if (val != 0 && val != 1)
+ return -EINVAL;
+
+ /*
+ * lock is not required, the btrfs_device struct can't be freed while
+ * its kobject btrfs_device::devid_kobj is still open.
+ */
+ device = container_of(kobj, struct btrfs_device, devid_kobj);
+
+ if (val &&
+ !test_bit(BTRFS_DEV_STATE_READ_PREFERRED, &device->dev_state)) {
+ set_bit(BTRFS_DEV_STATE_READ_PREFERRED, &device->dev_state);
+ btrfs_info(device->fs_devices->fs_info,
+ "set read preferred on devid %llu (%d)",
+ device->devid, task_pid_nr(current));
+ } else if (!val &&
+ test_bit(BTRFS_DEV_STATE_READ_PREFERRED, &device->dev_state)) {
+ clear_bit(BTRFS_DEV_STATE_READ_PREFERRED, &device->dev_state);
+ btrfs_info(device->fs_devices->fs_info,
+ "reset read preferred on devid %llu (%d)",
+ device->devid, task_pid_nr(current));
+ }
+
+ return len;
+}
+BTRFS_ATTR_RW(devid, read_preferred, btrfs_devinfo_read_pref_show,
+ btrfs_devinfo_read_pref_store);
+
static struct attribute *devid_attrs[] = {
BTRFS_ATTR_PTR(devid, in_fs_metadata),
BTRFS_ATTR_PTR(devid, missing),
BTRFS_ATTR_PTR(devid, replace_target),
BTRFS_ATTR_PTR(devid, writeable),
+ BTRFS_ATTR_PTR(devid, read_preferred),
NULL
};
ATTRIBUTE_GROUPS(devid);
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 71ba1f0e93f4..ea786864b903 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -51,6 +51,7 @@ struct btrfs_io_geometry {
#define BTRFS_DEV_STATE_REPLACE_TGT (3)
#define BTRFS_DEV_STATE_FLUSH_SENT (4)
#define BTRFS_DEV_STATE_NO_READA (5)
+#define BTRFS_DEV_STATE_READ_PREFERRED (6)
struct btrfs_zoned_device_info;
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v3 3/4] btrfs: introduce new read_policy device
2021-01-11 9:41 [PATCH v3 0/4] btrfs: read_policy types latency, device and round-robin Anand Jain
2021-01-11 9:41 ` [PATCH v3 1/4] btrfs: add read_policy latency Anand Jain
2021-01-11 9:41 ` [PATCH v3 2/4] btrfs: introduce new device-state read_preferred Anand Jain
@ 2021-01-11 9:41 ` Anand Jain
2021-01-19 19:44 ` Josef Bacik
2021-01-11 9:41 ` [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin Anand Jain
3 siblings, 1 reply; 15+ messages in thread
From: Anand Jain @ 2021-01-11 9:41 UTC (permalink / raw)
To: linux-btrfs; +Cc: dsterba, josef, Anand Jain
Read-policy type 'device' and device flag 'read-preferred':
The read-policy type device picks the device(s) flagged as
read-preferred for reading stripes of type raid1, raid10,
raid1c3 and raid1c4.
A system might contain SSD, nvme, iscsi, or san lun, and which are all
a non-rotational device, so it is not a good idea to set the read-preferred
automatically. Instead, device read-policy along with the read-preferred
flag provides an ability to do it manually. This advanced tuning is useful
in more than one situation, for example,
- In heterogeneous-disk volume, it provides an ability to manually choose
the low latency disks for reading.
- Useful for more accurate testing.
- Avoid known problematic device from reading the chunk until it is
replaced (by marking the other good devices as read-preferred).
Note:
If the read-policy type is set to 'device', but there isn't any device
which is flagged as read-preferred, then stripe 0 is used for reading.
The device replacement won't migrate the read-preferred flag to the new
replace the target device.
As of now, this is an in-memory only feature.
It's pointless to set the read-preferred flag on the missing device, as
IOs aren't submitted to the missing device.
If there is more than one read-preferred device in a chunk, the read IO
shall go to the stripe 0 as of now.
Usage example:
Consider a typical two disks raid1.
Configure devid1 for reading.
$ echo 1 > devinfo/1/read_preferred
$ cat devinfo/1/read_preferred
1
$ cat devinfo/2/read_preferred
0
$ pwd
/sys/fs/btrfs/12345678-1234-1234-1234-123456789abc
$ cat read_policy
[pid] device
$ echo device > ./read_policy
$ cat read_policy
pid [device]
Now read IOs are sent to devid 1 (sdb).
$ echo 3 > /proc/sys/vm/drop_caches
$ md5sum /btrfs/YkZI
$ iostat -zy 1 | egrep 'sdb|sdc' (from another terminal)
sdb 50.00 40048.00 0.00 40048 0
Change the read-preferred device from devid 1 to devid 2 (sdc).
$ echo 0 > ./devinfo/1/read_preferred
[ 3343.918658] BTRFS info (device sdb): reset read preferred on devid 1 (1334)
$ echo 1 > ./devinfo/2/read_preferred
[ 3343.919876] BTRFS info (device sdb): set read preferred on devid 2 (1334)
$ echo 3 > /proc/sys/vm/drop_caches
$ md5sum /btrfs/YkZI
Further read ios are sent to devid 2 (sdc).
$ iostat -zy 1 | egrep 'sdb|sdc' (from another terminal)
sdc 49.00 40048.00 0.00 40048 0
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v1:-
v2:-
v3: update the change log
fs/btrfs/sysfs.c | 3 ++-
fs/btrfs/volumes.c | 22 ++++++++++++++++++++++
fs/btrfs/volumes.h | 2 ++
3 files changed, 26 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index b5d6499fbad0..899b66c83db1 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -916,7 +916,8 @@ static bool strmatch(const char *buffer, const char *string)
}
/* Must follow the order as in enum btrfs_read_policy */
-static const char * const btrfs_read_policy_name[] = { "pid", "latency" };
+static const char * const btrfs_read_policy_name[] = { "pid", "latency",
+ "device" };
static ssize_t btrfs_read_policy_show(struct kobject *kobj,
struct kobj_attribute *a, char *buf)
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index f7a0a83d2cd4..50d4d54f7abd 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -5524,6 +5524,25 @@ static int btrfs_find_best_stripe(struct btrfs_fs_info *fs_info,
return best_stripe;
}
+static int btrfs_find_read_preferred(struct map_lookup *map, int first, int num_stripe)
+{
+ int stripe_index;
+ int last = first + num_stripe;
+
+ /*
+ * If there are more than one read preferred devices, then just pick the
+ * first found read preferred device as of now.
+ */
+ for (stripe_index = first; stripe_index < last; stripe_index++) {
+ if (test_bit(BTRFS_DEV_STATE_READ_PREFERRED,
+ &map->stripes[stripe_index].dev->dev_state))
+ return stripe_index;
+ }
+
+ /* If there is no read preferred device then just use the first stripe */
+ return first;
+}
+
static int find_live_mirror(struct btrfs_fs_info *fs_info,
struct map_lookup *map, int first,
int dev_replace_is_ongoing)
@@ -5557,6 +5576,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
preferred_mirror = btrfs_find_best_stripe(fs_info, map, first,
num_stripes);
break;
+ case BTRFS_READ_POLICY_DEVICE:
+ preferred_mirror = btrfs_find_read_preferred(map, first, num_stripes);
+ break;
}
if (dev_replace_is_ongoing &&
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index ea786864b903..8d5a2cddc0ab 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -225,6 +225,8 @@ enum btrfs_read_policy {
BTRFS_READ_POLICY_PID,
/* Find and use device with the lowest latency */
BTRFS_READ_POLICY_LATENCY,
+ /* Use the device marked with READ_PREFERRED state */
+ BTRFS_READ_POLICY_DEVICE,
BTRFS_NR_READ_POLICY,
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/4] btrfs: introduce new read_policy device
2021-01-11 9:41 ` [PATCH v3 3/4] btrfs: introduce new read_policy device Anand Jain
@ 2021-01-19 19:44 ` Josef Bacik
0 siblings, 0 replies; 15+ messages in thread
From: Josef Bacik @ 2021-01-19 19:44 UTC (permalink / raw)
To: Anand Jain, linux-btrfs; +Cc: dsterba
On 1/11/21 4:41 AM, Anand Jain wrote:
> Read-policy type 'device' and device flag 'read-preferred':
>
> The read-policy type device picks the device(s) flagged as
> read-preferred for reading stripes of type raid1, raid10,
> raid1c3 and raid1c4.
>
> A system might contain SSD, nvme, iscsi, or san lun, and which are all
> a non-rotational device, so it is not a good idea to set the read-preferred
> automatically. Instead, device read-policy along with the read-preferred
> flag provides an ability to do it manually. This advanced tuning is useful
> in more than one situation, for example,
> - In heterogeneous-disk volume, it provides an ability to manually choose
> the low latency disks for reading.
> - Useful for more accurate testing.
> - Avoid known problematic device from reading the chunk until it is
> replaced (by marking the other good devices as read-preferred).
>
> Note:
>
> If the read-policy type is set to 'device', but there isn't any device
> which is flagged as read-preferred, then stripe 0 is used for reading.
>
> The device replacement won't migrate the read-preferred flag to the new
> replace the target device.
>
> As of now, this is an in-memory only feature.
>
> It's pointless to set the read-preferred flag on the missing device, as
> IOs aren't submitted to the missing device.
>
> If there is more than one read-preferred device in a chunk, the read IO
> shall go to the stripe 0 as of now.
>
> Usage example:
>
> Consider a typical two disks raid1.
>
> Configure devid1 for reading.
>
> $ echo 1 > devinfo/1/read_preferred
> $ cat devinfo/1/read_preferred
> 1
> $ cat devinfo/2/read_preferred
> 0
>
> $ pwd
> /sys/fs/btrfs/12345678-1234-1234-1234-123456789abc
>
> $ cat read_policy
> [pid] device
> $ echo device > ./read_policy
> $ cat read_policy
> pid [device]
>
> Now read IOs are sent to devid 1 (sdb).
>
> $ echo 3 > /proc/sys/vm/drop_caches
> $ md5sum /btrfs/YkZI
>
> $ iostat -zy 1 | egrep 'sdb|sdc' (from another terminal)
> sdb 50.00 40048.00 0.00 40048 0
>
> Change the read-preferred device from devid 1 to devid 2 (sdc).
>
> $ echo 0 > ./devinfo/1/read_preferred
>
> [ 3343.918658] BTRFS info (device sdb): reset read preferred on devid 1 (1334)
>
> $ echo 1 > ./devinfo/2/read_preferred
>
> [ 3343.919876] BTRFS info (device sdb): set read preferred on devid 2 (1334)
>
> $ echo 3 > /proc/sys/vm/drop_caches
> $ md5sum /btrfs/YkZI
>
> Further read ios are sent to devid 2 (sdc).
>
> $ iostat -zy 1 | egrep 'sdb|sdc' (from another terminal)
> sdc 49.00 40048.00 0.00 40048 0
>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Thanks,
Josef
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin
2021-01-11 9:41 [PATCH v3 0/4] btrfs: read_policy types latency, device and round-robin Anand Jain
` (2 preceding siblings ...)
2021-01-11 9:41 ` [PATCH v3 3/4] btrfs: introduce new read_policy device Anand Jain
@ 2021-01-11 9:41 ` Anand Jain
2021-01-19 19:41 ` Josef Bacik
3 siblings, 1 reply; 15+ messages in thread
From: Anand Jain @ 2021-01-11 9:41 UTC (permalink / raw)
To: linux-btrfs; +Cc: dsterba, josef, Anand Jain
Add round-robin read policy to route the read IO to the next device in the
round-robin order. The chunk allocation and thus the stripe-index follows
the order of free space available on devices. So to make the round-robin
effective it shall follow the devid order instead of the stripe-index
order.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
--
RFC because: Provides terrible performance with the fio tests.
I am not yet sure if there is any io workload or a block layer
tuning that shall make this policy better. As of now just an
experimental patch.
fs/btrfs/sysfs.c | 2 +-
fs/btrfs/volumes.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/volumes.h | 3 +++
3 files changed, 54 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 899b66c83db1..d40b0ff054ca 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -917,7 +917,7 @@ static bool strmatch(const char *buffer, const char *string)
/* Must follow the order as in enum btrfs_read_policy */
static const char * const btrfs_read_policy_name[] = { "pid", "latency",
- "device" };
+ "device", "roundrobin" };
static ssize_t btrfs_read_policy_show(struct kobject *kobj,
struct kobj_attribute *a, char *buf)
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 50d4d54f7abd..60370b9121e0 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -5491,6 +5491,52 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
return ret;
}
+struct stripe_mirror {
+ u64 devid;
+ int map;
+};
+
+static int btrfs_cmp_devid(const void *a, const void *b)
+{
+ struct stripe_mirror *s1 = (struct stripe_mirror *)a;
+ struct stripe_mirror *s2 = (struct stripe_mirror *)b;
+
+ if (s1->devid < s2->devid)
+ return -1;
+ if (s1->devid > s2->devid)
+ return 1;
+ return 0;
+}
+
+static int btrfs_find_read_round_robin(struct map_lookup *map, int first,
+ int num_stripe)
+{
+ struct stripe_mirror stripes[4] = {0}; //4: for testing, works for now.
+ struct btrfs_fs_devices *fs_devices;
+ u64 devid;
+ int index, j, cnt;
+ int next_stripe;
+
+ index = 0;
+ for (j = first; j < first + num_stripe; j++) {
+ devid = map->stripes[j].dev->devid;
+
+ stripes[index].devid = devid;
+ stripes[index].map = j;
+
+ index++;
+ }
+
+ sort(stripes, num_stripe, sizeof(struct stripe_mirror),
+ btrfs_cmp_devid, NULL);
+
+ fs_devices = map->stripes[first].dev->fs_devices;
+ cnt = atomic_inc_return(&fs_devices->total_reads);
+ next_stripe = stripes[cnt % num_stripe].map;
+
+ return next_stripe;
+}
+
static int btrfs_find_best_stripe(struct btrfs_fs_info *fs_info,
struct map_lookup *map, int first,
int num_stripe)
@@ -5579,6 +5625,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
case BTRFS_READ_POLICY_DEVICE:
preferred_mirror = btrfs_find_read_preferred(map, first, num_stripes);
break;
+ case BTRFS_READ_POLICY_ROUND_ROBIN:
+ preferred_mirror = btrfs_find_read_round_robin(map, first,
+ num_stripes);
+ break;
}
if (dev_replace_is_ongoing &&
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 8d5a2cddc0ab..ce4490437f53 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -227,6 +227,8 @@ enum btrfs_read_policy {
BTRFS_READ_POLICY_LATENCY,
/* Use the device marked with READ_PREFERRED state */
BTRFS_READ_POLICY_DEVICE,
+ /* Distribute read IO equally across striped devices */
+ BTRFS_READ_POLICY_ROUND_ROBIN,
BTRFS_NR_READ_POLICY,
};
@@ -286,6 +288,7 @@ struct btrfs_fs_devices {
/* Policy used to read the mirrored stripes */
enum btrfs_read_policy read_policy;
+ atomic_t total_reads;
};
#define BTRFS_BIO_INLINE_CSUM_SIZE 64
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin
2021-01-11 9:41 ` [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin Anand Jain
@ 2021-01-19 19:41 ` Josef Bacik
2021-01-20 2:40 ` Anand Jain
0 siblings, 1 reply; 15+ messages in thread
From: Josef Bacik @ 2021-01-19 19:41 UTC (permalink / raw)
To: Anand Jain, linux-btrfs; +Cc: dsterba
On 1/11/21 4:41 AM, Anand Jain wrote:
> Add round-robin read policy to route the read IO to the next device in the
> round-robin order. The chunk allocation and thus the stripe-index follows
> the order of free space available on devices. So to make the round-robin
> effective it shall follow the devid order instead of the stripe-index
> order.
>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> --
> RFC because: Provides terrible performance with the fio tests.
> I am not yet sure if there is any io workload or a block layer
> tuning that shall make this policy better. As of now just an
> experimental patch.
>
Just drop this one, if we can't find a reason to use it then don't bother adding
the code. The other options have real world valuable uses, so stick with those.
Thanks,
Josef
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 4/4] btrfs: introduce new read_policy round-robin
2021-01-19 19:41 ` Josef Bacik
@ 2021-01-20 2:40 ` Anand Jain
0 siblings, 0 replies; 15+ messages in thread
From: Anand Jain @ 2021-01-20 2:40 UTC (permalink / raw)
To: Josef Bacik, linux-btrfs; +Cc: dsterba
On 20/1/21 3:41 am, Josef Bacik wrote:
> On 1/11/21 4:41 AM, Anand Jain wrote:
>> Add round-robin read policy to route the read IO to the next device in
>> the
>> round-robin order. The chunk allocation and thus the stripe-index follows
>> the order of free space available on devices. So to make the round-robin
>> effective it shall follow the devid order instead of the stripe-index
>> order.
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> --
>> RFC because: Provides terrible performance with the fio tests.
>> I am not yet sure if there is any io workload or a block layer
>> tuning that shall make this policy better. As of now just an
>> experimental patch.
>>
>
> Just drop this one, if we can't find a reason to use it then don't
> bother adding the code. The other options have real world valuable
> uses, so stick with those. Thanks,
>
Yep. I will drop this patch in the next iteration.
The low performance is attributed to the low number of read IO
mergers in the block layer. The consecutive blocks in my test case
(fio random read) were read from the other disk, so the block layer
lost the opportunity to merge the IOs.
Thanks, Anand
> Josef
>
^ permalink raw reply [flat|nested] 15+ messages in thread