Re: [PATCH 0/7][V11] btrfs: allocation_hint

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/2/16 02:49, Goffredo Baroncelli wrote:
> Hi Josef,
>
> gentle ping...
>
> few months ago you showed some interest in this patches set. Few of the
> cc-ed person use this patch set.
>
> I know that David showed some interest in the Anand approach (i.e. no
> knobs, but an automatic behavior looking at the speed of the devices).
>
> At the time when I tried this approach in the first attempts, I got the
> complain that the kernel may not know the performance differences of the
> disk (HDD vs SSD vs NVME vs ZONED disk...).

Sorry I didn't check the patches in details.

But I'm a little concerned about how to accurately determine the
performance of a device.

If doing it automatically, there must be some (commonly very short) time
spent to do the test.

In the very short time, I doubt we can even accurately got a full
picture of a device (from sequential read/write speed to IOPS values)

For spinning disks, the sequential read/write speed even change based on
their LBA address (as their physical location inside the plate can
change their linear velocity, since the angular velocity is fixed).

And even for SSD, IOPS can var dramatically due to cache/controller
difference.


For a proper performance aware setup, I guess the only correct way to
fetch performance characteristics is from the (advanced) user.

Or we may need to spent at least tens of minutes to do proper tests to
get the result.

For regular end users, the difference between SSD and HDD is huge enough
and simply preferring SSD for metadata is good enough.

But for more complex setup, like btrfs over LUKS over LVM (even crosses
several physical devices), I doubt if it's even possible to fetch the
correct performance characteristics automatically.

Thanks,
Qu
>
> Comments ?
>
> BR
> Goffredo
>
>
> On 26/01/2022 21.32, Goffredo Baroncelli wrote:
>> From: Goffredo Baroncelli <kreijack@inwind.it>
>>
>> Hi all,
>>
>> This patches set was born after some discussion between me, Zygo and
>> Josef.
>> Some details can be found in
>> https://github.com/btrfs/btrfs-todo/issues/19.
>>
>> Some further information about a real use case can be found in
>> https://lore.kernel.org/linux-btrfs/20210116002533.GE31381@hungrycats.org/
>>
>>
>> Recently Shafeeq told me that he is interested too, due to the
>> performance gain.
>>
>> In V8, revision I switched away from an ioctl API in favor of a sysfs
>> API (
>> see patch #2 and #3).
>>
>> In V9, I renamed the sysfs interface from devinfo/type to
>> devinfo/allocation_hint.
>> Moreover I renamed dev_info->type to dev_info->flags.
>>
>> In V10, I renamed the tag 'PREFERRED' from PREFERRED_X to X_PREFERRED;
>> I added
>> another devinfo property, called major_minor. For now it is unused;
>> the plan is to use this in btrfs-progs to go from the block device to
>> the filesystem.
>> First client will be "btrfs prop get/set allocation_hint", but I see
>> others possible
>> clients.
>> Finally I made some cleanup, and remove the mount option
>> 'allocation_hint'
>>
>> In V11 I added a new 'feature' file
>> /sys/fs/btrfs/features/allocation_hint
>> to help the detection of the allocation_hint feature.
>>
>> The idea behind this patches set, is to dedicate some disks (the
>> fastest one)
>> to the metadata chunk. My initial idea was a "soft" hint. However Zygo
>> asked an option for a "strong" hint (== mandatory). The result is that
>> each disk can be "tagged" by one of the following flags:
>> - BTRFS_DEV_ALLOCATION_METADATA_ONLY
>> - BTRFS_DEV_ALLOCATION_METADATA_PREFERRED
>> - BTRFS_DEV_ALLOCATION_DATA_PREFERRED
>> - BTRFS_DEV_ALLOCATION_DATA_ONLY
>>
>> When the chunk allocator search a disks to allocate a chunk, scans the
>> disks
>> in an order decided by these tags. For metadata, the order is:
>> *_METADATA_ONLY
>> *_METADATA_PREFERRED
>> *_DATA_PREFERRED
>>
>> The *_DATA_ONLY are not eligible from metadata chunk allocation.
>>
>> For the data chunk, the order is reversed, and the *_METADATA_ONLY are
>> excluded.
>>
>> The exact sort logic is to sort first for the "tag", and then for the
>> space
>> available. If there is no space available, the next "tag" disks set are
>> selected.
>>
>> To set these tags, a new property called "allocation_hint" was created.
>> There is a dedicated btrfs-prog patches set [[PATCH V11] btrfs-progs:
>> allocation_hint disk property].
>>
>> $ sudo mount /dev/loop0 /mnt/test-btrfs/
>> $ for i in /dev/loop[0-9]; do sudo ./btrfs prop get $i
>> allocation_hint; done
>> devid=1, path=/dev/loop0: allocation_hint=METADATA_PREFERRED
>> devid=2, path=/dev/loop1: allocation_hint=METADATA_PREFERRED
>> devid=3, path=/dev/loop2: allocation_hint=DATA_PREFERRED
>> devid=4, path=/dev/loop3: allocation_hint=DATA_PREFERRED
>> devid=5, path=/dev/loop4: allocation_hint=DATA_PREFERRED
>> devid=6, path=/dev/loop5: allocation_hint=DATA_ONLY
>> devid=7, path=/dev/loop6: allocation_hint=METADATA_ONLY
>> devid=8, path=/dev/loop7: allocation_hint=METADATA_ONLY
>>
>> $ sudo ./btrfs fi us /mnt/test-btrfs/
>> Overall:
>>      Device size:           2.75GiB
>>      Device allocated:           1.34GiB
>>      Device unallocated:           1.41GiB
>>      Device missing:             0.00B
>>      Used:             400.89MiB
>>      Free (estimated):           1.04GiB    (min: 1.04GiB)
>>      Data ratio:                  2.00
>>      Metadata ratio:              1.00
>>      Global reserve:           3.25MiB    (used: 0.00B)
>>      Multiple profiles:                no
>>
>> Data,RAID1: Size:542.00MiB, Used:200.25MiB (36.95%)
>>     /dev/loop0     288.00MiB
>>     /dev/loop1     288.00MiB
>>     /dev/loop2     127.00MiB
>>     /dev/loop3     127.00MiB
>>     /dev/loop4     127.00MiB
>>     /dev/loop5     127.00MiB
>>
>> Metadata,single: Size:256.00MiB, Used:384.00KiB (0.15%)
>>     /dev/loop1     256.00MiB
>>
>> System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
>>     /dev/loop0      32.00MiB
>>
>> Unallocated:
>>     /dev/loop0     704.00MiB
>>     /dev/loop1     480.00MiB
>>     /dev/loop2       1.00MiB
>>     /dev/loop3       1.00MiB
>>     /dev/loop4       1.00MiB
>>     /dev/loop5       1.00MiB
>>     /dev/loop6     128.00MiB
>>     /dev/loop7     128.00MiB
>>
>> # change the tag of some disks
>>
>> $ sudo ./btrfs prop set /dev/loop0 allocation_hint DATA_ONLY
>> $ sudo ./btrfs prop set /dev/loop1 allocation_hint DATA_ONLY
>> $ sudo ./btrfs prop set /dev/loop5 allocation_hint METADATA_ONLY
>>
>> $ for i in /dev/loop[0-9]; do sudo ./btrfs prop get $i
>> allocation_hint; done
>> devid=1, path=/dev/loop0: allocation_hint=DATA_ONLY
>> devid=2, path=/dev/loop1: allocation_hint=DATA_ONLY
>> devid=3, path=/dev/loop2: allocation_hint=DATA_PREFERRED
>> devid=4, path=/dev/loop3: allocation_hint=DATA_PREFERRED
>> devid=5, path=/dev/loop4: allocation_hint=DATA_PREFERRED
>> devid=6, path=/dev/loop5: allocation_hint=METADATA_ONLY
>> devid=7, path=/dev/loop6: allocation_hint=METADATA_ONLY
>> devid=8, path=/dev/loop7: allocation_hint=METADATA_ONLY
>>
>> $ sudo btrfs bal start --full-balance /mnt/test-btrfs/
>> $ sudo ./btrfs fi us /mnt/test-btrfs/
>> Overall:
>>      Device size:           2.75GiB
>>      Device allocated:         735.00MiB
>>      Device unallocated:           2.03GiB
>>      Device missing:             0.00B
>>      Used:             400.72MiB
>>      Free (estimated):           1.10GiB    (min: 1.10GiB)
>>      Data ratio:                  2.00
>>      Metadata ratio:              1.00
>>      Global reserve:           3.25MiB    (used: 0.00B)
>>      Multiple profiles:                no
>>
>> Data,RAID1: Size:288.00MiB, Used:200.19MiB (69.51%)
>>     /dev/loop0     288.00MiB
>>     /dev/loop1     288.00MiB
>>
>> Metadata,single: Size:127.00MiB, Used:336.00KiB (0.26%)
>>     /dev/loop5     127.00MiB
>>
>> System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
>>     /dev/loop7      32.00MiB
>>
>> Unallocated:
>>     /dev/loop0     736.00MiB
>>     /dev/loop1     736.00MiB
>>     /dev/loop2     128.00MiB
>>     /dev/loop3     128.00MiB
>>     /dev/loop4     128.00MiB
>>     /dev/loop5       1.00MiB
>>     /dev/loop6     128.00MiB
>>     /dev/loop7      96.00MiB
>>
>>
>> #As you can see all the metadata were placed on the disk loop5/loop7
>> even if
>> #the most empty one are loop0 and loop1.
>>
>>
>>
>> Furher works:
>> - the tool which show the space available should consider the tagging (eg
>>    the disks tagged by _METADATA_ONLY should be excluded from the data
>>    availability)
>> - allow btrfs-prog to change the allocation_hint even when the filesystem
>>    is not mounted.
>>
>> In following emails, there will be the btrfs-progs patches set and the
>> xfstest
>> suite patches set.
>>
>>
>> Comments are welcome
>> BR
>> G.Baroncelli
>>
>> Revision:
>> V11:
>> - added the property /sys/fs/btrfs/features/allocation_hint
>>
>> V10:
>> - renamed two disk tags/constants:
>>          - *_METADATA_PREFERRED -> *_METADATA_PREFERRED
>>          - *_DATA_PREFERRED -> *_DATA_EFERRED
>> - add /sys/fs/btrfs/$UUID/devinfo/$DEVID/major_minor
>> - revise some commit description
>> - refactored the code of gather_device_info(): one portion of this code
>>    is moved in a separate function called sort_and_reduce_device_info()
>>    for clarity.
>> - removed the 'allocation_hint' mount option
>>
>> V9:
>> - rename dev_item->type to dev_item->flags
>> - rename /sys/fs/btrfs/$UUID/devinfo/type -> allocation_hint
>>
>> V8:
>> - drop the ioctl API, instead use a sysfs one
>>
>> V7:
>> - make more room in the struct btrfs_ioctl_dev_properties up to 1K
>> - leave in btrfs_tree.h only the costants
>> - removed the mount option (sic)
>> - correct an 'use before check' in the while loop (signaled
>>    by Zygo)
>> - add a 2nd sort to be sure that the device_info array is in the
>>    expected order
>>
>> V6:
>> - add further values to the hints: add the possibility to
>>    exclude a disk for a chunk type
>>
>>
>>
>> Goffredo Baroncelli (7):
>>    btrfs: add flags to give an hint to the chunk allocator
>>    btrfs: export the device allocation_hint property in sysfs
>>    btrfs: change the device allocation_hint property via sysfs
>>    btrfs: add allocation_hint mode
>>    btrfs: rename dev_item->type to dev_item->flags
>>    btrfs: add major and minor to sysfs
>>    Add /sys/fs/btrfs/features/allocation_hint
>>
>>   fs/btrfs/ctree.h                |   4 +-
>>   fs/btrfs/disk-io.c              |   2 +-
>>   fs/btrfs/sysfs.c                | 105 +++++++++++++++++++++++++++
>>   fs/btrfs/volumes.c              | 121 ++++++++++++++++++++++++++++++--
>>   fs/btrfs/volumes.h              |   7 +-
>>   include/uapi/linux/btrfs_tree.h |  20 +++++-
>>   6 files changed, 245 insertions(+), 14 deletions(-)
>>
>
>