Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums
@ 2019-10-07  9:11 Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
                   ` (4 more replies)
  0 siblings, 5 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

This series adds support for two additional checksum algorithms to btrfs. These
algorithms are xxhash64[1] and sha256[2].

xxhash64 is a fast non-cryptographic hash function with good collision resistance.
It has a constant output length of 8 Byte (64 Bit), it provides a good
trade-off between collision resistance and speed compared to the currently
used crc32c.

sha256 is the 32 Byte (256 Bit) variant of the SHA-2 cryptographic hash. It
provides cryptographically secure collision resistance with a trade off in
speed.

Support for xxhash64 in mkfs.btrfs is in the current devel branch and sha256
support will be sent separately after this patch-set.

In addition to adding these two hash algorithms two sysfs files are
implemented, one being /sys/fs/btrfs/features/supported_checksums showing the
in kernel support for different checksumming algorithms. The other one is
/sys/fs/btrfs/$FSID/checksum showing the checksum used for a specific
file-system and the used in-kernel driver for this checksum.

Here is an example in a qemu vm:
host:/# cat /sys/fs/btrfs/features/supported_checksums
crc32c, xxhash64, sha256
host:/# cat /sys/fs/btrfs/3cf09516-5bb8-498f-834d-e9ec54043546/checksum
sha256 (sha256-generic)

This series has survived the usual regression testing with xfstests.

I could not observe any performance differences between any of these hashes in
my test setup 256K mixed read-write IO to a single file from a single process
on both a 5700rpm SATA 3G Disk behind a HPE SmartArray RAID HBA and RAM Disk.

Here's the raw numbers for the spinning rust behind SATA:
CRC32C Buffered Read (KiB/s): Avg: 7881, Min: 7495, Max: 8744, Stdev: 508
CRC32C Buffered Write (KiB/s): Avg: 7883, Min: 7497, Max: 8746, Stdev: 508
                               
CRC32C Direct Read (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
CRC32C Direct Write (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7

XXHASH64 Buffered Read (KiB/s): Avg: 8143, Min: 7748, Max: 8721, Stdev: 355
XXHASH64 Buffered Write (KiB/s): Avg: 8145, Min: 7750, Max: 8722, Stdev: 355

XXHASH64 Direct Read (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
XXHASH64 Direct Write (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
                               
SHA256 Buffered Read (KiB/s): Avg: 7997, Min: 7665, Max: 8336, Stdev: 273
SHA256 Buffered Write (KiB/s): Avg: 7998, Min: 7666, Max: 8337, Stdev: 273

SHA256 Direct Read (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
SHA256 Direct Write (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36

The reason I could not observe any changes in performance is the fact that the
btrfs checksumming process takes only 0.04% of the IO path. This also explains
the very small standard deviation in the above table as I stooped benchmarking
after 5 benchmark runs.

The hottest call chain (according to perf) is this:

17.08%     0.00%  kworker/u128:9-  [kernel.vmlinux]  [k] btrfs_finish_ordered_io
 |
 ---btrfs_finish_ordered_io
    |          
     --17.04%--insert_reserved_file_extent.constprop.75
       |          
        --17.02%--__btrfs_drop_extents
     	   |          
     	    --16.94%--btrfs_free_extent
     		|          
     		 --16.94%--btrfs_add_delayed_data_ref
     		   |          
     		    --16.90%--btrfs_qgroup_trace_extent_post
     			      btrfs_find_all_roots
     			      |          
     			       --16.90%--btrfs_find_all_roots_safe
     				 |          
     				  --16.89%--find_parent_nodes
     				    |          
     				     --16.68%--resolve_indirect_refs
					       [snip]

[1] https://cyan4973.github.io/xxHash
[2] https://en.wikipedia.org/wiki/SHA-2

David Sterba (1):
  btrfs: sysfs: export supported checksums

Johannes Thumshirn (3):
  btrfs: add xxhash64 to checksumming algorithms
  btrfs: add sha256 to checksumming algorithms
  btrfs: show used checksum driver per filesystem in sysfs

 fs/btrfs/Kconfig                |  2 ++
 fs/btrfs/ctree.c                |  7 ++++++
 fs/btrfs/ctree.h                |  2 ++
 fs/btrfs/disk-io.c              |  2 ++
 fs/btrfs/super.c                |  2 ++
 fs/btrfs/sysfs.c                | 48 +++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/btrfs_tree.h |  2 ++
 7 files changed, 65 insertions(+)

-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Add xxhash64 to the list of possible checksumming algorithms used by
BTRFS.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/Kconfig                | 1 +
 fs/btrfs/ctree.c                | 1 +
 fs/btrfs/disk-io.c              | 1 +
 fs/btrfs/super.c                | 1 +
 include/uapi/linux/btrfs_tree.h | 1 +
 5 files changed, 5 insertions(+)

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 38651fae7f21..6d5a01c57da3 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -5,6 +5,7 @@ config BTRFS_FS
 	select CRYPTO
 	select CRYPTO_CRC32C
 	select LIBCRC32C
+	select CRYPTO_XXHASH
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index 3a4d8e27e565..a1b317cf193b 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -34,6 +34,7 @@ static const struct btrfs_csums {
 	const char	*name;
 } btrfs_csums[] = {
 	[BTRFS_CSUM_TYPE_CRC32] = { .size = 4, .name = "crc32c" },
+	[BTRFS_CSUM_TYPE_XXHASH] = { .size = 8, .name = "xxhash64" },
 };
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index ea40898d49eb..b0a6904b6139 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -352,6 +352,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
 {
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
 		return true;
 	default:
 		return false;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index d5d15a19f51d..c91cdf123cd8 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2462,3 +2462,4 @@ module_exit(exit_btrfs_fs)
 
 MODULE_LICENSE("GPL");
 MODULE_SOFTDEP("pre: crc32c");
+MODULE_SOFTDEP("pre: xxhash64");
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index b65c7ee75bc7..ba2f125a3a1c 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -302,6 +302,7 @@
 /* csum types */
 enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
 };
 
 /*
-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 2/4] btrfs: add sha256 to checksumming algorithms
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
@ 2019-10-07  9:11 ` " Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Add sha256 to the list of possible checksumming algorithms used by BTRFS.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/Kconfig                | 1 +
 fs/btrfs/ctree.c                | 1 +
 fs/btrfs/disk-io.c              | 1 +
 fs/btrfs/super.c                | 1 +
 include/uapi/linux/btrfs_tree.h | 1 +
 5 files changed, 5 insertions(+)

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 6d5a01c57da3..75b6d10c9845 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -6,6 +6,7 @@ config BTRFS_FS
 	select CRYPTO_CRC32C
 	select LIBCRC32C
 	select CRYPTO_XXHASH
+	select CRYPTO_SHA256
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index a1b317cf193b..b66509ee62eb 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -35,6 +35,7 @@ static const struct btrfs_csums {
 } btrfs_csums[] = {
 	[BTRFS_CSUM_TYPE_CRC32] = { .size = 4, .name = "crc32c" },
 	[BTRFS_CSUM_TYPE_XXHASH] = { .size = 8, .name = "xxhash64" },
+	[BTRFS_CSUM_TYPE_SHA256] = { .size = 32, .name = "sha256" },
 };
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index b0a6904b6139..99feff25bbef 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -353,6 +353,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
 	case BTRFS_CSUM_TYPE_XXHASH:
+	case BTRFS_CSUM_TYPE_SHA256:
 		return true;
 	default:
 		return false;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index c91cdf123cd8..36440336c08b 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2463,3 +2463,4 @@ module_exit(exit_btrfs_fs)
 MODULE_LICENSE("GPL");
 MODULE_SOFTDEP("pre: crc32c");
 MODULE_SOFTDEP("pre: xxhash64");
+MODULE_SOFTDEP("pre: sha256");
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index ba2f125a3a1c..838cfd0af5c3 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -303,6 +303,7 @@
 enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32	= 0,
 	BTRFS_CSUM_TYPE_XXHASH	= 1,
+	BTRFS_CSUM_TYPE_SHA256	= 2,
 };
 
 /*
-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07 15:36   ` Nikolay Borisov
  2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
  2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov
  4 siblings, 1 reply; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

From: David Sterba <dsterba@suse.com>

Export supported checksum algorithms via sysfs.

Co-developed-by: David Sterba <dsterba@suse.com> 
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/ctree.c |  5 +++++
 fs/btrfs/ctree.h |  2 ++
 fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+)

diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index b66509ee62eb..5debd74dc61c 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
 	return btrfs_csums[csum_type].name;
 }
 
+size_t btrfs_get_num_csums(void)
+{
+	return ARRAY_SIZE(btrfs_csums);
+}
+
 struct btrfs_path *btrfs_alloc_path(void)
 {
 	return kmem_cache_zalloc(btrfs_path_cachep, GFP_NOFS);
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index d17e79a40930..0180554f6970 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2165,6 +2165,8 @@ BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s);
 const char *btrfs_super_csum_name(u16 csum_type);
+size_t btrfs_get_num_csums(void);
+
 
 /*
  * The leaf data grows from end-to-front in the node.
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index f6d3c80f2e28..aeebbdfe1a98 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -246,6 +246,28 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
 	return mode;
 }
 
+static ssize_t btrfs_supported_checksums_show(struct kobject *kobj,
+					      struct kobj_attribute *a,
+					      char *buf)
+{
+	ssize_t ret = 0;
+	int i;
+
+	for (i = 0; i < btrfs_get_num_csums(); i++) {
+		/*
+		 * This "trick" only works as long as 'enum btrfs_csum_type' has
+		 * no holes in it
+		 */
+		ret += snprintf(buf + ret, PAGE_SIZE - ret, "%s%s",
+				(i == 0 ? "" : ", "),
+				btrfs_super_csum_name(i));
+
+	}
+
+	ret += snprintf(buf + ret, PAGE_SIZE - ret, "\n");
+	return ret;
+}
+
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
 BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
@@ -259,6 +281,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
 BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
 BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
 
+static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
+	.kobj_attr = __INIT_KOBJ_ATTR(supported_checksums, S_IRUGO,
+				      btrfs_supported_checksums_show,
+				      NULL),
+	.feature_set	= FEAT_INCOMPAT,
+	.feature_bit	= 0,
+};
+
 static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(mixed_backref),
 	BTRFS_FEAT_ATTR_PTR(default_subvol),
@@ -272,6 +302,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(no_holes),
 	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
 	BTRFS_FEAT_ATTR_PTR(free_space_tree),
+
+	&btrfs_attr_features_checksums_name.kobj_attr.attr,
+
 	NULL
 };
 
-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
                   ` (2 preceding siblings ...)
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Show the used driver for the checksum algorithm for the filesystem in
sysfs.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/sysfs.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index aeebbdfe1a98..11a3cf7f563e 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -9,6 +9,7 @@
 #include <linux/spinlock.h>
 #include <linux/completion.h>
 #include <linux/bug.h>
+#include <crypto/hash.h>
 
 #include "ctree.h"
 #include "disk-io.h"
@@ -637,6 +638,19 @@ static ssize_t btrfs_metadata_uuid_show(struct kobject *kobj,
 
 BTRFS_ATTR(, metadata_uuid, btrfs_metadata_uuid_show);
 
+static ssize_t btrfs_checksum_show(struct kobject *kobj,
+				   struct kobj_attribute *a, char *buf)
+{
+	struct btrfs_fs_info *fs_info = to_fs_info(kobj);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
+	return snprintf(buf, PAGE_SIZE, "%s (%s)\n",
+			btrfs_super_csum_name(csum_type),
+			crypto_shash_driver_name(fs_info->csum_shash));
+}
+
+BTRFS_ATTR(, checksum, btrfs_checksum_show);
+
 static const struct attribute *btrfs_attrs[] = {
 	BTRFS_ATTR_PTR(, label),
 	BTRFS_ATTR_PTR(, nodesize),
@@ -644,6 +658,7 @@ static const struct attribute *btrfs_attrs[] = {
 	BTRFS_ATTR_PTR(, clone_alignment),
 	BTRFS_ATTR_PTR(, quota_override),
 	BTRFS_ATTR_PTR(, metadata_uuid),
+	BTRFS_ATTR_PTR(, checksum),
 	NULL,
 };
 
-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
                   ` (3 preceding siblings ...)
  2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
@ 2019-10-07 14:58 ` Nikolay Borisov
  4 siblings, 0 replies; 10+ messages in thread
From: Nikolay Borisov @ 2019-10-07 14:58 UTC (permalink / raw)
  To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist



On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> This series adds support for two additional checksum algorithms to btrfs. These
> algorithms are xxhash64[1] and sha256[2].
> 
> xxhash64 is a fast non-cryptographic hash function with good collision resistance.
> It has a constant output length of 8 Byte (64 Bit), it provides a good
> trade-off between collision resistance and speed compared to the currently
> used crc32c.
> 
> sha256 is the 32 Byte (256 Bit) variant of the SHA-2 cryptographic hash. It
> provides cryptographically secure collision resistance with a trade off in
> speed.
> 
> Support for xxhash64 in mkfs.btrfs is in the current devel branch and sha256
> support will be sent separately after this patch-set.
> 
> In addition to adding these two hash algorithms two sysfs files are
> implemented, one being /sys/fs/btrfs/features/supported_checksums showing the
> in kernel support for different checksumming algorithms. The other one is
> /sys/fs/btrfs/$FSID/checksum showing the checksum used for a specific
> file-system and the used in-kernel driver for this checksum.
> 
> Here is an example in a qemu vm:
> host:/# cat /sys/fs/btrfs/features/supported_checksums
> crc32c, xxhash64, sha256
> host:/# cat /sys/fs/btrfs/3cf09516-5bb8-498f-834d-e9ec54043546/checksum
> sha256 (sha256-generic)
> 
> This series has survived the usual regression testing with xfstests.
> 
> I could not observe any performance differences between any of these hashes in
> my test setup 256K mixed read-write IO to a single file from a single process
> on both a 5700rpm SATA 3G Disk behind a HPE SmartArray RAID HBA and RAM Disk.
> 
> Here's the raw numbers for the spinning rust behind SATA:
> CRC32C Buffered Read (KiB/s): Avg: 7881, Min: 7495, Max: 8744, Stdev: 508
> CRC32C Buffered Write (KiB/s): Avg: 7883, Min: 7497, Max: 8746, Stdev: 508
>                                
> CRC32C Direct Read (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
> CRC32C Direct Write (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
> 
> XXHASH64 Buffered Read (KiB/s): Avg: 8143, Min: 7748, Max: 8721, Stdev: 355
> XXHASH64 Buffered Write (KiB/s): Avg: 8145, Min: 7750, Max: 8722, Stdev: 355
> 
> XXHASH64 Direct Read (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
> XXHASH64 Direct Write (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
>                                
> SHA256 Buffered Read (KiB/s): Avg: 7997, Min: 7665, Max: 8336, Stdev: 273
> SHA256 Buffered Write (KiB/s): Avg: 7998, Min: 7666, Max: 8337, Stdev: 273
> 
> SHA256 Direct Read (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
> SHA256 Direct Write (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
> 
> The reason I could not observe any changes in performance is the fact that the
> btrfs checksumming process takes only 0.04% of the IO path. This also explains
> the very small standard deviation in the above table as I stooped benchmarking
> after 5 benchmark runs.
> 
> The hottest call chain (according to perf) is this:
> 
> 17.08%     0.00%  kworker/u128:9-  [kernel.vmlinux]  [k] btrfs_finish_ordered_io
>  |
>  ---btrfs_finish_ordered_io
>     |          
>      --17.04%--insert_reserved_file_extent.constprop.75
>        |          
>         --17.02%--__btrfs_drop_extents
>      	   |          
>      	    --16.94%--btrfs_free_extent
>      		|          
>      		 --16.94%--btrfs_add_delayed_data_ref
>      		   |          
>      		    --16.90%--btrfs_qgroup_trace_extent_post

Yeah, we know qgroups tracing and backrefs resolv are somewhat slow. How
about benchmarking without them since I believe this to be more
representative of how people use btrfs.

>      			      btrfs_find_all_roots
>      			      |          
>      			       --16.90%--btrfs_find_all_roots_safe
>      				 |          
>      				  --16.89%--find_parent_nodes
>      				    |          
>      				     --16.68%--resolve_indirect_refs
> 					       [snip]
> 
> [1] https://cyan4973.github.io/xxHash
> [2] https://en.wikipedia.org/wiki/SHA-2
> 
> David Sterba (1):
>   btrfs: sysfs: export supported checksums
> 
> Johannes Thumshirn (3):
>   btrfs: add xxhash64 to checksumming algorithms
>   btrfs: add sha256 to checksumming algorithms
>   btrfs: show used checksum driver per filesystem in sysfs
> 
>  fs/btrfs/Kconfig                |  2 ++
>  fs/btrfs/ctree.c                |  7 ++++++
>  fs/btrfs/ctree.h                |  2 ++
>  fs/btrfs/disk-io.c              |  2 ++
>  fs/btrfs/super.c                |  2 ++
>  fs/btrfs/sysfs.c                | 48 +++++++++++++++++++++++++++++++++++++++++
>  include/uapi/linux/btrfs_tree.h |  2 ++
>  7 files changed, 65 insertions(+)
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
@ 2019-10-07 15:36   ` Nikolay Borisov
  2019-10-07 15:46     ` David Sterba
  0 siblings, 1 reply; 10+ messages in thread
From: Nikolay Borisov @ 2019-10-07 15:36 UTC (permalink / raw)
  To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist



On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> Export supported checksum algorithms via sysfs.
> 
> Co-developed-by: David Sterba <dsterba@suse.com> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/ctree.c |  5 +++++
>  fs/btrfs/ctree.h |  2 ++
>  fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
>  3 files changed, 40 insertions(+)
> 
> diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
> index b66509ee62eb..5debd74dc61c 100644
> --- a/fs/btrfs/ctree.c
> +++ b/fs/btrfs/ctree.c
> @@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
>  	return btrfs_csums[csum_type].name;
>  }
>  
> +size_t btrfs_get_num_csums(void)
> +{
> +	return ARRAY_SIZE(btrfs_csums);
> +}

nit: This function is used only once and the ARRAY_SIZE() macro is
descriptive enough, why not just remove it and opencoude the call to
array_size

> +
>  struct btrfs_path *btrfs_alloc_path(void)
>  {
>  	return kmem_cache_zalloc(btrfs_path_cachep, GFP_NOFS);
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index d17e79a40930..0180554f6970 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -2165,6 +2165,8 @@ BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
>  
>  int btrfs_super_csum_size(const struct btrfs_super_block *s);
>  const char *btrfs_super_csum_name(u16 csum_type);
> +size_t btrfs_get_num_csums(void);
> +
>  
>  /*
>   * The leaf data grows from end-to-front in the node.
> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> index f6d3c80f2e28..aeebbdfe1a98 100644
> --- a/fs/btrfs/sysfs.c
> +++ b/fs/btrfs/sysfs.c
> @@ -246,6 +246,28 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
>  	return mode;
>  }
>  
> +static ssize_t btrfs_supported_checksums_show(struct kobject *kobj,
> +					      struct kobj_attribute *a,
> +					      char *buf)
> +{
> +	ssize_t ret = 0;
> +	int i;
> +
> +	for (i = 0; i < btrfs_get_num_csums(); i++) {
> +		/*
> +		 * This "trick" only works as long as 'enum btrfs_csum_type' has
> +		 * no holes in it
> +		 */
> +		ret += snprintf(buf + ret, PAGE_SIZE - ret, "%s%s",
> +				(i == 0 ? "" : ", "),
> +				btrfs_super_csum_name(i));
> +
> +	}
> +
> +	ret += snprintf(buf + ret, PAGE_SIZE - ret, "\n");
> +	return ret;
> +}
> +
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
>  BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
> @@ -259,6 +281,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
>  BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
>  BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
>  
> +static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
> +	.kobj_attr = __INIT_KOBJ_ATTR(supported_checksums, S_IRUGO,
> +				      btrfs_supported_checksums_show,
> +				      NULL),
> +	.feature_set	= FEAT_INCOMPAT,
> +	.feature_bit	= 0,
> +};
> +
>  static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(mixed_backref),
>  	BTRFS_FEAT_ATTR_PTR(default_subvol),
> @@ -272,6 +302,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(no_holes),
>  	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
>  	BTRFS_FEAT_ATTR_PTR(free_space_tree),
> +
> +	&btrfs_attr_features_checksums_name.kobj_attr.attr,
> +
>  	NULL
>  };
>  
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07 15:36   ` Nikolay Borisov
@ 2019-10-07 15:46     ` David Sterba
  2019-10-08  6:47       ` Johannes Thumshirn
  0 siblings, 1 reply; 10+ messages in thread
From: David Sterba @ 2019-10-07 15:46 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist

On Mon, Oct 07, 2019 at 06:36:14PM +0300, Nikolay Borisov wrote:
> 
> 
> On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> > From: David Sterba <dsterba@suse.com>
> > 
> > Export supported checksum algorithms via sysfs.
> > 
> > Co-developed-by: David Sterba <dsterba@suse.com> 
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  fs/btrfs/ctree.c |  5 +++++
> >  fs/btrfs/ctree.h |  2 ++
> >  fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
> >  3 files changed, 40 insertions(+)
> > 
> > diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
> > index b66509ee62eb..5debd74dc61c 100644
> > --- a/fs/btrfs/ctree.c
> > +++ b/fs/btrfs/ctree.c
> > @@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
> >  	return btrfs_csums[csum_type].name;
> >  }
> >  
> > +size_t btrfs_get_num_csums(void)
> > +{
> > +	return ARRAY_SIZE(btrfs_csums);
> > +}
> 
> nit: This function is used only once and the ARRAY_SIZE() macro is
> descriptive enough, why not just remove it and opencoude the call to
> array_size

Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07 15:46     ` David Sterba
@ 2019-10-08  6:47       ` Johannes Thumshirn
  2019-10-08 11:40         ` David Sterba
  0 siblings, 1 reply; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-08  6:47 UTC (permalink / raw)
  To: dsterba, Nikolay Borisov, David Sterba, Linux BTRFS Mailinglist

On 07/10/2019 17:46, David Sterba wrote:
[...]
>> nit: This function is used only once and the ARRAY_SIZE() macro is
>> descriptive enough, why not just remove it and opencoude the call to
>> array_size
> 
> Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.

Nope, btrfs_csums[] is defined in ctree.c, so I can't get the size of
this array outside of ctree.c. And it was moved to ctree.c from ctree.h
on request by David.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-08  6:47       ` Johannes Thumshirn
@ 2019-10-08 11:40         ` David Sterba
  0 siblings, 0 replies; 10+ messages in thread
From: David Sterba @ 2019-10-08 11:40 UTC (permalink / raw)
  To: Johannes Thumshirn
  Cc: dsterba, Nikolay Borisov, David Sterba, Linux BTRFS Mailinglist

On Tue, Oct 08, 2019 at 08:47:14AM +0200, Johannes Thumshirn wrote:
> On 07/10/2019 17:46, David Sterba wrote:
> [...]
> >> nit: This function is used only once and the ARRAY_SIZE() macro is
> >> descriptive enough, why not just remove it and opencoude the call to
> >> array_size
> > 
> > Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.
> 
> Nope, btrfs_csums[] is defined in ctree.c, so I can't get the size of
> this array outside of ctree.c. And it was moved to ctree.c from ctree.h
> on request by David.

Ohh, right you are of course.

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
2019-10-07 15:36   ` Nikolay Borisov
2019-10-07 15:46     ` David Sterba
2019-10-08  6:47       ` Johannes Thumshirn
2019-10-08 11:40         ` David Sterba
2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox