All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums
@ 2019-10-07  9:11 Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
                   ` (4 more replies)
  0 siblings, 5 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

This series adds support for two additional checksum algorithms to btrfs. These
algorithms are xxhash64[1] and sha256[2].

xxhash64 is a fast non-cryptographic hash function with good collision resistance.
It has a constant output length of 8 Byte (64 Bit), it provides a good
trade-off between collision resistance and speed compared to the currently
used crc32c.

sha256 is the 32 Byte (256 Bit) variant of the SHA-2 cryptographic hash. It
provides cryptographically secure collision resistance with a trade off in
speed.

Support for xxhash64 in mkfs.btrfs is in the current devel branch and sha256
support will be sent separately after this patch-set.

In addition to adding these two hash algorithms two sysfs files are
implemented, one being /sys/fs/btrfs/features/supported_checksums showing the
in kernel support for different checksumming algorithms. The other one is
/sys/fs/btrfs/$FSID/checksum showing the checksum used for a specific
file-system and the used in-kernel driver for this checksum.

Here is an example in a qemu vm:
host:/# cat /sys/fs/btrfs/features/supported_checksums
crc32c, xxhash64, sha256
host:/# cat /sys/fs/btrfs/3cf09516-5bb8-498f-834d-e9ec54043546/checksum
sha256 (sha256-generic)

This series has survived the usual regression testing with xfstests.

I could not observe any performance differences between any of these hashes in
my test setup 256K mixed read-write IO to a single file from a single process
on both a 5700rpm SATA 3G Disk behind a HPE SmartArray RAID HBA and RAM Disk.

Here's the raw numbers for the spinning rust behind SATA:
CRC32C Buffered Read (KiB/s): Avg: 7881, Min: 7495, Max: 8744, Stdev: 508
CRC32C Buffered Write (KiB/s): Avg: 7883, Min: 7497, Max: 8746, Stdev: 508
                               
CRC32C Direct Read (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
CRC32C Direct Write (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7

XXHASH64 Buffered Read (KiB/s): Avg: 8143, Min: 7748, Max: 8721, Stdev: 355
XXHASH64 Buffered Write (KiB/s): Avg: 8145, Min: 7750, Max: 8722, Stdev: 355

XXHASH64 Direct Read (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
XXHASH64 Direct Write (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
                               
SHA256 Buffered Read (KiB/s): Avg: 7997, Min: 7665, Max: 8336, Stdev: 273
SHA256 Buffered Write (KiB/s): Avg: 7998, Min: 7666, Max: 8337, Stdev: 273

SHA256 Direct Read (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
SHA256 Direct Write (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36

The reason I could not observe any changes in performance is the fact that the
btrfs checksumming process takes only 0.04% of the IO path. This also explains
the very small standard deviation in the above table as I stooped benchmarking
after 5 benchmark runs.

The hottest call chain (according to perf) is this:

17.08%     0.00%  kworker/u128:9-  [kernel.vmlinux]  [k] btrfs_finish_ordered_io
 |
 ---btrfs_finish_ordered_io
    |          
     --17.04%--insert_reserved_file_extent.constprop.75
       |          
        --17.02%--__btrfs_drop_extents
     	   |          
     	    --16.94%--btrfs_free_extent
     		|          
     		 --16.94%--btrfs_add_delayed_data_ref
     		   |          
     		    --16.90%--btrfs_qgroup_trace_extent_post
     			      btrfs_find_all_roots
     			      |          
     			       --16.90%--btrfs_find_all_roots_safe
     				 |          
     				  --16.89%--find_parent_nodes
     				    |          
     				     --16.68%--resolve_indirect_refs
					       [snip]

[1] https://cyan4973.github.io/xxHash
[2] https://en.wikipedia.org/wiki/SHA-2

David Sterba (1):
  btrfs: sysfs: export supported checksums

Johannes Thumshirn (3):
  btrfs: add xxhash64 to checksumming algorithms
  btrfs: add sha256 to checksumming algorithms
  btrfs: show used checksum driver per filesystem in sysfs

 fs/btrfs/Kconfig                |  2 ++
 fs/btrfs/ctree.c                |  7 ++++++
 fs/btrfs/ctree.h                |  2 ++
 fs/btrfs/disk-io.c              |  2 ++
 fs/btrfs/super.c                |  2 ++
 fs/btrfs/sysfs.c                | 48 +++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/btrfs_tree.h |  2 ++
 7 files changed, 65 insertions(+)

-- 
2.16.4


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Add xxhash64 to the list of possible checksumming algorithms used by
BTRFS.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/Kconfig                | 1 +
 fs/btrfs/ctree.c                | 1 +
 fs/btrfs/disk-io.c              | 1 +
 fs/btrfs/super.c                | 1 +
 include/uapi/linux/btrfs_tree.h | 1 +
 5 files changed, 5 insertions(+)

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 38651fae7f21..6d5a01c57da3 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -5,6 +5,7 @@ config BTRFS_FS
 	select CRYPTO
 	select CRYPTO_CRC32C
 	select LIBCRC32C
+	select CRYPTO_XXHASH
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index 3a4d8e27e565..a1b317cf193b 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -34,6 +34,7 @@ static const struct btrfs_csums {
 	const char	*name;
 } btrfs_csums[] = {
 	[BTRFS_CSUM_TYPE_CRC32] = { .size = 4, .name = "crc32c" },
+	[BTRFS_CSUM_TYPE_XXHASH] = { .size = 8, .name = "xxhash64" },
 };
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index ea40898d49eb..b0a6904b6139 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -352,6 +352,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
 {
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
 		return true;
 	default:
 		return false;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index d5d15a19f51d..c91cdf123cd8 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2462,3 +2462,4 @@ module_exit(exit_btrfs_fs)
 
 MODULE_LICENSE("GPL");
 MODULE_SOFTDEP("pre: crc32c");
+MODULE_SOFTDEP("pre: xxhash64");
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index b65c7ee75bc7..ba2f125a3a1c 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -302,6 +302,7 @@
 /* csum types */
 enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
 };
 
 /*
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH 2/4] btrfs: add sha256 to checksumming algorithms
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Add sha256 to the list of possible checksumming algorithms used by BTRFS.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/Kconfig                | 1 +
 fs/btrfs/ctree.c                | 1 +
 fs/btrfs/disk-io.c              | 1 +
 fs/btrfs/super.c                | 1 +
 include/uapi/linux/btrfs_tree.h | 1 +
 5 files changed, 5 insertions(+)

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 6d5a01c57da3..75b6d10c9845 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -6,6 +6,7 @@ config BTRFS_FS
 	select CRYPTO_CRC32C
 	select LIBCRC32C
 	select CRYPTO_XXHASH
+	select CRYPTO_SHA256
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index a1b317cf193b..b66509ee62eb 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -35,6 +35,7 @@ static const struct btrfs_csums {
 } btrfs_csums[] = {
 	[BTRFS_CSUM_TYPE_CRC32] = { .size = 4, .name = "crc32c" },
 	[BTRFS_CSUM_TYPE_XXHASH] = { .size = 8, .name = "xxhash64" },
+	[BTRFS_CSUM_TYPE_SHA256] = { .size = 32, .name = "sha256" },
 };
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index b0a6904b6139..99feff25bbef 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -353,6 +353,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
 	case BTRFS_CSUM_TYPE_XXHASH:
+	case BTRFS_CSUM_TYPE_SHA256:
 		return true;
 	default:
 		return false;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index c91cdf123cd8..36440336c08b 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2463,3 +2463,4 @@ module_exit(exit_btrfs_fs)
 MODULE_LICENSE("GPL");
 MODULE_SOFTDEP("pre: crc32c");
 MODULE_SOFTDEP("pre: xxhash64");
+MODULE_SOFTDEP("pre: sha256");
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index ba2f125a3a1c..838cfd0af5c3 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -303,6 +303,7 @@
 enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32	= 0,
 	BTRFS_CSUM_TYPE_XXHASH	= 1,
+	BTRFS_CSUM_TYPE_SHA256	= 2,
 };
 
 /*
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
  2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07 15:36   ` Nikolay Borisov
  2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
  2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov
  4 siblings, 1 reply; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

From: David Sterba <dsterba@suse.com>

Export supported checksum algorithms via sysfs.

Co-developed-by: David Sterba <dsterba@suse.com> 
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/ctree.c |  5 +++++
 fs/btrfs/ctree.h |  2 ++
 fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+)

diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index b66509ee62eb..5debd74dc61c 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
 	return btrfs_csums[csum_type].name;
 }
 
+size_t btrfs_get_num_csums(void)
+{
+	return ARRAY_SIZE(btrfs_csums);
+}
+
 struct btrfs_path *btrfs_alloc_path(void)
 {
 	return kmem_cache_zalloc(btrfs_path_cachep, GFP_NOFS);
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index d17e79a40930..0180554f6970 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2165,6 +2165,8 @@ BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
 
 int btrfs_super_csum_size(const struct btrfs_super_block *s);
 const char *btrfs_super_csum_name(u16 csum_type);
+size_t btrfs_get_num_csums(void);
+
 
 /*
  * The leaf data grows from end-to-front in the node.
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index f6d3c80f2e28..aeebbdfe1a98 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -246,6 +246,28 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
 	return mode;
 }
 
+static ssize_t btrfs_supported_checksums_show(struct kobject *kobj,
+					      struct kobj_attribute *a,
+					      char *buf)
+{
+	ssize_t ret = 0;
+	int i;
+
+	for (i = 0; i < btrfs_get_num_csums(); i++) {
+		/*
+		 * This "trick" only works as long as 'enum btrfs_csum_type' has
+		 * no holes in it
+		 */
+		ret += snprintf(buf + ret, PAGE_SIZE - ret, "%s%s",
+				(i == 0 ? "" : ", "),
+				btrfs_super_csum_name(i));
+
+	}
+
+	ret += snprintf(buf + ret, PAGE_SIZE - ret, "\n");
+	return ret;
+}
+
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
 BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
@@ -259,6 +281,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
 BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
 BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
 
+static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
+	.kobj_attr = __INIT_KOBJ_ATTR(supported_checksums, S_IRUGO,
+				      btrfs_supported_checksums_show,
+				      NULL),
+	.feature_set	= FEAT_INCOMPAT,
+	.feature_bit	= 0,
+};
+
 static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(mixed_backref),
 	BTRFS_FEAT_ATTR_PTR(default_subvol),
@@ -272,6 +302,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(no_holes),
 	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
 	BTRFS_FEAT_ATTR_PTR(free_space_tree),
+
+	&btrfs_attr_features_checksums_name.kobj_attr.attr,
+
 	NULL
 };
 
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
                   ` (2 preceding siblings ...)
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
@ 2019-10-07  9:11 ` Johannes Thumshirn
  2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov
  4 siblings, 0 replies; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-07  9:11 UTC (permalink / raw)
  To: David Sterba; +Cc: Linux BTRFS Mailinglist, Nikolay Borisov, Johannes Thumshirn

Show the used driver for the checksum algorithm for the filesystem in
sysfs.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/sysfs.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index aeebbdfe1a98..11a3cf7f563e 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -9,6 +9,7 @@
 #include <linux/spinlock.h>
 #include <linux/completion.h>
 #include <linux/bug.h>
+#include <crypto/hash.h>
 
 #include "ctree.h"
 #include "disk-io.h"
@@ -637,6 +638,19 @@ static ssize_t btrfs_metadata_uuid_show(struct kobject *kobj,
 
 BTRFS_ATTR(, metadata_uuid, btrfs_metadata_uuid_show);
 
+static ssize_t btrfs_checksum_show(struct kobject *kobj,
+				   struct kobj_attribute *a, char *buf)
+{
+	struct btrfs_fs_info *fs_info = to_fs_info(kobj);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
+	return snprintf(buf, PAGE_SIZE, "%s (%s)\n",
+			btrfs_super_csum_name(csum_type),
+			crypto_shash_driver_name(fs_info->csum_shash));
+}
+
+BTRFS_ATTR(, checksum, btrfs_checksum_show);
+
 static const struct attribute *btrfs_attrs[] = {
 	BTRFS_ATTR_PTR(, label),
 	BTRFS_ATTR_PTR(, nodesize),
@@ -644,6 +658,7 @@ static const struct attribute *btrfs_attrs[] = {
 	BTRFS_ATTR_PTR(, clone_alignment),
 	BTRFS_ATTR_PTR(, quota_override),
 	BTRFS_ATTR_PTR(, metadata_uuid),
+	BTRFS_ATTR_PTR(, checksum),
 	NULL,
 };
 
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums
  2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
                   ` (3 preceding siblings ...)
  2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
@ 2019-10-07 14:58 ` Nikolay Borisov
  4 siblings, 0 replies; 10+ messages in thread
From: Nikolay Borisov @ 2019-10-07 14:58 UTC (permalink / raw)
  To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist



On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> This series adds support for two additional checksum algorithms to btrfs. These
> algorithms are xxhash64[1] and sha256[2].
> 
> xxhash64 is a fast non-cryptographic hash function with good collision resistance.
> It has a constant output length of 8 Byte (64 Bit), it provides a good
> trade-off between collision resistance and speed compared to the currently
> used crc32c.
> 
> sha256 is the 32 Byte (256 Bit) variant of the SHA-2 cryptographic hash. It
> provides cryptographically secure collision resistance with a trade off in
> speed.
> 
> Support for xxhash64 in mkfs.btrfs is in the current devel branch and sha256
> support will be sent separately after this patch-set.
> 
> In addition to adding these two hash algorithms two sysfs files are
> implemented, one being /sys/fs/btrfs/features/supported_checksums showing the
> in kernel support for different checksumming algorithms. The other one is
> /sys/fs/btrfs/$FSID/checksum showing the checksum used for a specific
> file-system and the used in-kernel driver for this checksum.
> 
> Here is an example in a qemu vm:
> host:/# cat /sys/fs/btrfs/features/supported_checksums
> crc32c, xxhash64, sha256
> host:/# cat /sys/fs/btrfs/3cf09516-5bb8-498f-834d-e9ec54043546/checksum
> sha256 (sha256-generic)
> 
> This series has survived the usual regression testing with xfstests.
> 
> I could not observe any performance differences between any of these hashes in
> my test setup 256K mixed read-write IO to a single file from a single process
> on both a 5700rpm SATA 3G Disk behind a HPE SmartArray RAID HBA and RAM Disk.
> 
> Here's the raw numbers for the spinning rust behind SATA:
> CRC32C Buffered Read (KiB/s): Avg: 7881, Min: 7495, Max: 8744, Stdev: 508
> CRC32C Buffered Write (KiB/s): Avg: 7883, Min: 7497, Max: 8746, Stdev: 508
>                                
> CRC32C Direct Read (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
> CRC32C Direct Write (KiB/s): Avg: 331, Min: 319, Max: 339, Stdev: 7
> 
> XXHASH64 Buffered Read (KiB/s): Avg: 8143, Min: 7748, Max: 8721, Stdev: 355
> XXHASH64 Buffered Write (KiB/s): Avg: 8145, Min: 7750, Max: 8722, Stdev: 355
> 
> XXHASH64 Direct Read (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
> XXHASH64 Direct Write (KiB/s): Avg: 311, Min: 248, Max: 336, Stdev: 36
>                                
> SHA256 Buffered Read (KiB/s): Avg: 7997, Min: 7665, Max: 8336, Stdev: 273
> SHA256 Buffered Write (KiB/s): Avg: 7998, Min: 7666, Max: 8337, Stdev: 273
> 
> SHA256 Direct Read (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
> SHA256 Direct Write (KiB/s): Avg: 312, Min: 248, Max: 336, Stdev: 36
> 
> The reason I could not observe any changes in performance is the fact that the
> btrfs checksumming process takes only 0.04% of the IO path. This also explains
> the very small standard deviation in the above table as I stooped benchmarking
> after 5 benchmark runs.
> 
> The hottest call chain (according to perf) is this:
> 
> 17.08%     0.00%  kworker/u128:9-  [kernel.vmlinux]  [k] btrfs_finish_ordered_io
>  |
>  ---btrfs_finish_ordered_io
>     |          
>      --17.04%--insert_reserved_file_extent.constprop.75
>        |          
>         --17.02%--__btrfs_drop_extents
>      	   |          
>      	    --16.94%--btrfs_free_extent
>      		|          
>      		 --16.94%--btrfs_add_delayed_data_ref
>      		   |          
>      		    --16.90%--btrfs_qgroup_trace_extent_post

Yeah, we know qgroups tracing and backrefs resolv are somewhat slow. How
about benchmarking without them since I believe this to be more
representative of how people use btrfs.

>      			      btrfs_find_all_roots
>      			      |          
>      			       --16.90%--btrfs_find_all_roots_safe
>      				 |          
>      				  --16.89%--find_parent_nodes
>      				    |          
>      				     --16.68%--resolve_indirect_refs
> 					       [snip]
> 
> [1] https://cyan4973.github.io/xxHash
> [2] https://en.wikipedia.org/wiki/SHA-2
> 
> David Sterba (1):
>   btrfs: sysfs: export supported checksums
> 
> Johannes Thumshirn (3):
>   btrfs: add xxhash64 to checksumming algorithms
>   btrfs: add sha256 to checksumming algorithms
>   btrfs: show used checksum driver per filesystem in sysfs
> 
>  fs/btrfs/Kconfig                |  2 ++
>  fs/btrfs/ctree.c                |  7 ++++++
>  fs/btrfs/ctree.h                |  2 ++
>  fs/btrfs/disk-io.c              |  2 ++
>  fs/btrfs/super.c                |  2 ++
>  fs/btrfs/sysfs.c                | 48 +++++++++++++++++++++++++++++++++++++++++
>  include/uapi/linux/btrfs_tree.h |  2 ++
>  7 files changed, 65 insertions(+)
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
@ 2019-10-07 15:36   ` Nikolay Borisov
  2019-10-07 15:46     ` David Sterba
  0 siblings, 1 reply; 10+ messages in thread
From: Nikolay Borisov @ 2019-10-07 15:36 UTC (permalink / raw)
  To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist



On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> Export supported checksum algorithms via sysfs.
> 
> Co-developed-by: David Sterba <dsterba@suse.com> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/ctree.c |  5 +++++
>  fs/btrfs/ctree.h |  2 ++
>  fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
>  3 files changed, 40 insertions(+)
> 
> diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
> index b66509ee62eb..5debd74dc61c 100644
> --- a/fs/btrfs/ctree.c
> +++ b/fs/btrfs/ctree.c
> @@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
>  	return btrfs_csums[csum_type].name;
>  }
>  
> +size_t btrfs_get_num_csums(void)
> +{
> +	return ARRAY_SIZE(btrfs_csums);
> +}

nit: This function is used only once and the ARRAY_SIZE() macro is
descriptive enough, why not just remove it and opencoude the call to
array_size

> +
>  struct btrfs_path *btrfs_alloc_path(void)
>  {
>  	return kmem_cache_zalloc(btrfs_path_cachep, GFP_NOFS);
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index d17e79a40930..0180554f6970 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -2165,6 +2165,8 @@ BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
>  
>  int btrfs_super_csum_size(const struct btrfs_super_block *s);
>  const char *btrfs_super_csum_name(u16 csum_type);
> +size_t btrfs_get_num_csums(void);
> +
>  
>  /*
>   * The leaf data grows from end-to-front in the node.
> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> index f6d3c80f2e28..aeebbdfe1a98 100644
> --- a/fs/btrfs/sysfs.c
> +++ b/fs/btrfs/sysfs.c
> @@ -246,6 +246,28 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
>  	return mode;
>  }
>  
> +static ssize_t btrfs_supported_checksums_show(struct kobject *kobj,
> +					      struct kobj_attribute *a,
> +					      char *buf)
> +{
> +	ssize_t ret = 0;
> +	int i;
> +
> +	for (i = 0; i < btrfs_get_num_csums(); i++) {
> +		/*
> +		 * This "trick" only works as long as 'enum btrfs_csum_type' has
> +		 * no holes in it
> +		 */
> +		ret += snprintf(buf + ret, PAGE_SIZE - ret, "%s%s",
> +				(i == 0 ? "" : ", "),
> +				btrfs_super_csum_name(i));
> +
> +	}
> +
> +	ret += snprintf(buf + ret, PAGE_SIZE - ret, "\n");
> +	return ret;
> +}
> +
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
>  BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
> @@ -259,6 +281,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
>  BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
>  BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
>  
> +static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
> +	.kobj_attr = __INIT_KOBJ_ATTR(supported_checksums, S_IRUGO,
> +				      btrfs_supported_checksums_show,
> +				      NULL),
> +	.feature_set	= FEAT_INCOMPAT,
> +	.feature_bit	= 0,
> +};
> +
>  static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(mixed_backref),
>  	BTRFS_FEAT_ATTR_PTR(default_subvol),
> @@ -272,6 +302,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(no_holes),
>  	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
>  	BTRFS_FEAT_ATTR_PTR(free_space_tree),
> +
> +	&btrfs_attr_features_checksums_name.kobj_attr.attr,
> +
>  	NULL
>  };
>  
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07 15:36   ` Nikolay Borisov
@ 2019-10-07 15:46     ` David Sterba
  2019-10-08  6:47       ` Johannes Thumshirn
  0 siblings, 1 reply; 10+ messages in thread
From: David Sterba @ 2019-10-07 15:46 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist

On Mon, Oct 07, 2019 at 06:36:14PM +0300, Nikolay Borisov wrote:
> 
> 
> On 7.10.19 г. 12:11 ч., Johannes Thumshirn wrote:
> > From: David Sterba <dsterba@suse.com>
> > 
> > Export supported checksum algorithms via sysfs.
> > 
> > Co-developed-by: David Sterba <dsterba@suse.com> 
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  fs/btrfs/ctree.c |  5 +++++
> >  fs/btrfs/ctree.h |  2 ++
> >  fs/btrfs/sysfs.c | 33 +++++++++++++++++++++++++++++++++
> >  3 files changed, 40 insertions(+)
> > 
> > diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
> > index b66509ee62eb..5debd74dc61c 100644
> > --- a/fs/btrfs/ctree.c
> > +++ b/fs/btrfs/ctree.c
> > @@ -53,6 +53,11 @@ const char *btrfs_super_csum_name(u16 csum_type)
> >  	return btrfs_csums[csum_type].name;
> >  }
> >  
> > +size_t btrfs_get_num_csums(void)
> > +{
> > +	return ARRAY_SIZE(btrfs_csums);
> > +}
> 
> nit: This function is used only once and the ARRAY_SIZE() macro is
> descriptive enough, why not just remove it and opencoude the call to
> array_size

Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-07 15:46     ` David Sterba
@ 2019-10-08  6:47       ` Johannes Thumshirn
  2019-10-08 11:40         ` David Sterba
  0 siblings, 1 reply; 10+ messages in thread
From: Johannes Thumshirn @ 2019-10-08  6:47 UTC (permalink / raw)
  To: dsterba, Nikolay Borisov, David Sterba, Linux BTRFS Mailinglist

On 07/10/2019 17:46, David Sterba wrote:
[...]
>> nit: This function is used only once and the ARRAY_SIZE() macro is
>> descriptive enough, why not just remove it and opencoude the call to
>> array_size
> 
> Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.

Nope, btrfs_csums[] is defined in ctree.c, so I can't get the size of
this array outside of ctree.c. And it was moved to ctree.c from ctree.h
on request by David.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 3/4] btrfs: sysfs: export supported checksums
  2019-10-08  6:47       ` Johannes Thumshirn
@ 2019-10-08 11:40         ` David Sterba
  0 siblings, 0 replies; 10+ messages in thread
From: David Sterba @ 2019-10-08 11:40 UTC (permalink / raw)
  To: Johannes Thumshirn
  Cc: dsterba, Nikolay Borisov, David Sterba, Linux BTRFS Mailinglist

On Tue, Oct 08, 2019 at 08:47:14AM +0200, Johannes Thumshirn wrote:
> On 07/10/2019 17:46, David Sterba wrote:
> [...]
> >> nit: This function is used only once and the ARRAY_SIZE() macro is
> >> descriptive enough, why not just remove it and opencoude the call to
> >> array_size
> > 
> > Agreed, ARRAY_SIZE in loops is fine, it's a compile-time constant.
> 
> Nope, btrfs_csums[] is defined in ctree.c, so I can't get the size of
> this array outside of ctree.c. And it was moved to ctree.c from ctree.h
> on request by David.

Ohh, right you are of course.

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-10-08 11:39 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-07  9:11 [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 1/4] btrfs: add xxhash64 to checksumming algorithms Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 2/4] btrfs: add sha256 " Johannes Thumshirn
2019-10-07  9:11 ` [PATCH 3/4] btrfs: sysfs: export supported checksums Johannes Thumshirn
2019-10-07 15:36   ` Nikolay Borisov
2019-10-07 15:46     ` David Sterba
2019-10-08  6:47       ` Johannes Thumshirn
2019-10-08 11:40         ` David Sterba
2019-10-07  9:11 ` [PATCH 4/4] btrfs: show used checksum driver per filesystem in sysfs Johannes Thumshirn
2019-10-07 14:58 ` [PATCH 0/4] Add xxhash64 and sha256 as possible new checksums Nikolay Borisov

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.