[PATCH v5 0/7] btrfs-progs: support xxhash64 checksums

[PATCH v5 0/7] btrfs-progs: support xxhash64 checksums

[Johannes Thumshirn]

Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
prepared for an easy addition of new checksums, this patchset implements
XXHASH64 as a second, fast but not cryptographically secure checksum hash.

This patchset is fully bisectible and available on github at
https://github.com/morbidrsa/btrfs-progs/tree/mkfs-xxhash64.v5

Changes since v4:
- Rebased onto latest 'devel' branch and dropped applied changes
- Split 'btrfs-progs: add xxhash64 as checksum algorithm' into several atomic
  patches
- Changed test code to using 'TEST_ENABLE_OVERRIDE'

Johannes Thumshirn (7):
  btrfs-progs: add option for checksum type to mkfs
  btrfs-progs: add is_valid_csum_type() helper
  btrfs-progs: add table for checksum type and name
  btrfs-progs: also print checksum type when running mkfs
  btrfs-progs: add xxhash64 to mkfs
  btrfs-progs: move crc32c implementation to crypto/
  btrfs-progs: add test override for mkfs to use different checksums

 Android.mk                                  |  4 ++--
 Makefile                                    |  7 ++++---
 btrfs-crc.c                                 |  2 +-
 btrfs-find-root.c                           |  2 +-
 btrfs-sb-mod.c                              |  2 +-
 btrfs.c                                     |  2 +-
 cmds/inspect-dump-super.c                   | 26 ++++++++++++++++----------
 cmds/rescue-chunk-recover.c                 |  2 +-
 cmds/rescue-super-recover.c                 |  2 +-
 common/utils.c                              |  2 +-
 convert/common.c                            |  2 +-
 convert/main.c                              |  6 +++---
 {kernel-lib => crypto}/crc32c.c             |  2 +-
 {kernel-lib => crypto}/crc32c.h             |  0
 crypto/hash.c                               | 17 +++++++++++++++++
 crypto/hash.h                               | 10 ++++++++++
 ctree.c                                     | 28 ++++++++++++++++++++++++++++
 ctree.h                                     | 18 +++++++-----------
 disk-io.c                                   |  9 ++++++---
 extent-tree.c                               |  2 +-
 file-item.c                                 |  2 +-
 free-space-cache.c                          |  2 +-
 hash.h                                      |  2 +-
 image/main.c                                |  7 ++++---
 image/sanitize.c                            |  2 +-
 library-test.c                              |  2 +-
 mkfs/common.c                               | 14 +++++++-------
 mkfs/main.c                                 | 10 ++++++++--
 send-stream.c                               |  2 +-
 tests/common                                | 10 ++++++++--
 tests/mkfs-tests/001-basic-profiles/test.sh |  8 +++++++-
 31 files changed, 143 insertions(+), 63 deletions(-)
 rename {kernel-lib => crypto}/crc32c.c (99%)
 rename {kernel-lib => crypto}/crc32c.h (100%)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h

-- 
2.16.4

[PATCH v5 1/7] btrfs-progs: add option for checksum type to mkfs

[Johannes Thumshirn]

Add an option to mkfs to specify which checksum algorithm will be used for
the filesystem.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 mkfs/main.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mkfs/main.c b/mkfs/main.c
index 7a7b6417a947..61184c8e7525 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -354,6 +354,7 @@ static void print_usage(int ret)
 	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
 	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
 	printf("\t-f|--force              force overwrite of existing filesystem\n");
+	printf("\t--checksum              checksum algorithm to use (default: crc32c)\n");
 	printf("  general:\n");
 	printf("\t-q|--quiet              no messages except errors\n");
 	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
-- 
2.16.4

Re: [PATCH v5 1/7] btrfs-progs: add option for checksum type to mkfs

[Johannes Thumshirn]

On 25/09/2019 15:37, Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  mkfs/main.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 7a7b6417a947..61184c8e7525 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -354,6 +354,7 @@ static void print_usage(int ret)
>  	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
>  	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
>  	printf("\t-f|--force              force overwrite of existing filesystem\n");
> +	printf("\t--checksum              checksum algorithm to use (default: crc32c)\n");
>  	printf("  general:\n");
>  	printf("\t-q|--quiet              no messages except errors\n");
>  	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
> 

This looks like it's a rebasing error. I'll investigate what went wrong
here.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH v5 1/7] btrfs-progs: add option for checksum type to mkfs

[Johannes Thumshirn]

On 25/09/2019 15:50, Johannes Thumshirn wrote:
[...]
> This looks like it's a rebasing error. I'll investigate what went wrong
> here.

Yup that's a rebase error, please ignore this patch


-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH v5 1/7] btrfs-progs: add option for checksum type to mkfs

[David Sterba]

On Wed, Sep 25, 2019 at 04:01:48PM +0200, Johannes Thumshirn wrote:
> On 25/09/2019 15:50, Johannes Thumshirn wrote:
> [...]
> > This looks like it's a rebasing error. I'll investigate what went wrong
> > here.
> 
> Yup that's a rebase error, please ignore this patch

No problem. I moved the option to the 'features' sections, so the rebase
on your side kept it in the 'creation' section.

[PATCH v5 2/7] btrfs-progs: add is_valid_csum_type() helper

[Johannes Thumshirn]

Add a helper to check if we have a valid csum type from the super block.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 cmds/inspect-dump-super.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 58bf82b0bbd3..272df1bfdc14 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -311,6 +311,16 @@ static void print_readable_super_flag(u64 flag)
 				     super_flags_num, BTRFS_SUPER_FLAG_SUPP);
 }
 
+static bool is_valid_csum_type(u16 csum_type)
+{
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void dump_superblock(struct btrfs_super_block *sb, int full)
 {
 	int i;
@@ -326,7 +336,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	csum_type = btrfs_super_csum_type(sb);
 	csum_size = BTRFS_CSUM_SIZE;
 	printf("csum_type\t\t%hu (", csum_type);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (!is_valid_csum_type(csum_type)) {
 		printf("INVALID");
 	} else {
 		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
@@ -342,8 +352,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	printf("csum\t\t\t0x");
 	for (i = 0, p = sb->csum; i < csum_size; i++)
 		printf("%02x", p[i]);
-	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
-	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
+	if (!is_valid_csum_type(csum_type))
 		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
 	else if (check_csum_sblock(sb, csum_size, csum_type))
 		printf(" [match]");
-- 
2.16.4

[PATCH v5 3/7] btrfs-progs: add table for checksum type and name

[Johannes Thumshirn]

Adding this table will make extending btrfs-progs with new checksum types
easier.

Also add accessor functions to access the table fields.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 cmds/inspect-dump-super.c |  8 ++------
 convert/common.c          |  2 +-
 convert/main.c            |  4 ++--
 ctree.c                   | 28 ++++++++++++++++++++++++++++
 ctree.h                   | 15 +++++----------
 disk-io.c                 |  4 ++--
 image/main.c              |  5 +++--
 mkfs/common.c             | 14 +++++++-------
 8 files changed, 50 insertions(+), 30 deletions(-)

diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 272df1bfdc14..bf380ad2b56a 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -339,12 +339,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	if (!is_valid_csum_type(csum_type)) {
 		printf("INVALID");
 	} else {
-		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
-			printf("crc32c");
-			csum_size = btrfs_csum_sizes[csum_type];
-		} else {
-			printf("unknown");
-		}
+		printf("%s", btrfs_super_csum_name(csum_type));
+		csum_size = btrfs_super_csum_size(sb);
 	}
 	printf(")\n");
 	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
diff --git a/convert/common.c b/convert/common.c
index 2e2318a5863e..3cb2d9d44881 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
 {
 	int ret;
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 
 	/* Temporary extent buffer is always mapped 1:1 on disk */
diff --git a/convert/main.c b/convert/main.c
index 611ca7693bde..bb689be9f3e4 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,8 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
 	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
 	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
-			     btrfs_super_csum_type(super));
+	csum_tree_block_size(buf, btrfs_super_csum_size(super),
+			     0, btrfs_super_csum_type(super));
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 		BTRFS_SUPER_INFO_OFFSET);
 	if (ret != BTRFS_SUPER_INFO_SIZE)
diff --git a/ctree.c b/ctree.c
index c97fbb40bb6e..6058c9b660b0 100644
--- a/ctree.c
+++ b/ctree.c
@@ -38,6 +38,34 @@ static int balance_node_right(struct btrfs_trans_handle *trans,
 			      struct extent_buffer *dst_buf,
 			      struct extent_buffer *src_buf);
 
+static struct btrfs_csum {
+	u16 size;
+	const char *name;
+} btrfs_csums[] = {
+	[BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+};
+
+u16 btrfs_super_csum_size(const struct btrfs_super_block *sb)
+{
+	u16 csum_type = btrfs_super_csum_type(sb);
+	return btrfs_csums[csum_type].size;
+}
+
+const char *btrfs_super_csum_name(u16 csum_type)
+{
+	return btrfs_csums[csum_type].name;
+}
+
+size_t btrfs_super_num_csums(void)
+{
+	return ARRAY_SIZE(btrfs_csums);
+}
+
+u16 btrfs_csum_type_size(u16 csum_type)
+{
+	return btrfs_csums[csum_type].size;
+}
+
 inline void btrfs_init_path(struct btrfs_path *p)
 {
 	memset(p, 0, sizeof(*p));
diff --git a/ctree.h b/ctree.h
index e0dfb4853b18..f70271dc658e 100644
--- a/ctree.h
+++ b/ctree.h
@@ -169,9 +169,6 @@ enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32   = 0,
 };
 
-/* four bytes for CRC32 */
-static int btrfs_csum_sizes[] = { 4 };
-
 #define BTRFS_EMPTY_DIR_SIZE 0
 
 #define BTRFS_FT_UNKNOWN	0
@@ -2263,13 +2260,6 @@ BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
 			 uuid_tree_generation, 64);
 BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
 
-static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
-{
-	int t = btrfs_super_csum_type(s);
-	BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
-	return btrfs_csum_sizes[t];
-}
-
 static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
 {
 	return offsetof(struct btrfs_leaf, items);
@@ -2700,6 +2690,11 @@ void btrfs_set_item_key_unsafe(struct btrfs_root *root,
 			       struct btrfs_path *path,
 			       struct btrfs_key *new_key);
 
+u16 btrfs_super_csum_size(const struct btrfs_super_block *s);
+const char *btrfs_super_csum_name(u16 csum_type);
+u16 btrfs_csum_type_size(u16 csum_type);
+size_t btrfs_super_num_csums(void);
+
 /* root-item.c */
 int btrfs_add_root_ref(struct btrfs_trans_handle *trans,
 		       struct btrfs_root *tree_root,
diff --git a/disk-io.c b/disk-io.c
index 20a6569d259c..72c672919cf9 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -1377,11 +1377,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	}
 
 	csum_type = btrfs_super_csum_type(sb);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (csum_type >= btrfs_super_num_csums()) {
 		error("unsupported checksum algorithm %u", csum_type);
 		return -EIO;
 	}
-	csum_size = btrfs_csum_sizes[csum_type];
+	csum_size = btrfs_super_csum_size(sb);
 
 	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
 			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
diff --git a/image/main.c b/image/main.c
index 0c8ffede56f5..f713e47b85cb 100644
--- a/image/main.c
+++ b/image/main.c
@@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
 
 static void csum_block(u8 *buf, size_t len)
 {
-	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
+	u16 csum_size = btrfs_csum_type_size(BTRFS_CSUM_TYPE_CRC32);
+	u8 result[csum_size];
 	u32 crc = ~(u32)0;
 	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
 	put_unaligned_le32(~crc, result);
-	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
+	memcpy(buf, result, csum_size);
 }
 
 static int has_name(struct btrfs_key *key)
diff --git a/mkfs/common.c b/mkfs/common.c
index 4a417bd7a306..469b88d6a8d3 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 
 	/* write back root tree */
@@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
@@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
@@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
@@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
@@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
@@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
-- 
2.16.4

[PATCH v5 4/7] btrfs-progs: also print checksum type when running mkfs

[Johannes Thumshirn]

As mkfs will grow new checksums, print the used checksum in it's versbose
output.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 mkfs/main.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index 61184c8e7525..04509e788a52 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -1380,7 +1380,9 @@ raid_groups:
 			pretty_size(allocation.system));
 		printf("SSD detected:       %s\n", ssd ? "yes" : "no");
 		btrfs_parse_features_to_string(features_buf, features);
-		printf("Incompat features:  %s", features_buf);
+		printf("Incompat features:  %s\n", features_buf);
+		printf("Checksum:           %s",
+		       btrfs_super_csum_name(mkfs_cfg.csum_type));
 		printf("\n");
 
 		list_all_devices(root);
-- 
2.16.4

[PATCH v5 5/7] btrfs-progs: add xxhash64 to mkfs

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 Makefile                  |  3 ++-
 cmds/inspect-dump-super.c |  1 +
 crypto/hash.c             | 17 +++++++++++++++++
 crypto/hash.h             | 10 ++++++++++
 ctree.h                   |  3 ++-
 disk-io.c                 |  3 +++
 mkfs/main.c               |  3 +++
 7 files changed, 38 insertions(+), 2 deletions(-)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h

diff --git a/Makefile b/Makefile
index 370e0c37ff65..45530749e2b9 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
 	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
 		   kernel-lib/crc32c.o common/messages.o \
-		   uuid-tree.o utils-lib.o common/rbtree-utils.o
+		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
+		   crypto/hash.o crypto/xxhash.o
 libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
 	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
 	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index bf380ad2b56a..73e986ed8ee8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -315,6 +315,7 @@ static bool is_valid_csum_type(u16 csum_type)
 {
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
 		return true;
 	default:
 		return false;
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..58a3890bd467
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,17 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+	XXH64_hash_t hash;
+
+	hash = XXH64(buf, length, 0);
+	/*
+	 * NOTE: we're not taking the canonical form here but the plain hash to
+	 * be compatible with the kernel implementation!
+	 */
+	memcpy(out, &hash, 8);
+
+	return 0;
+}
+
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX	32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/ctree.h b/ctree.h
index f70271dc658e..144c89eb4a36 100644
--- a/ctree.h
+++ b/ctree.h
@@ -166,7 +166,8 @@ struct btrfs_free_space_ctl;
 
 /* csum types */
 enum btrfs_csum_type {
-	BTRFS_CSUM_TYPE_CRC32   = 0,
+	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
 };
 
 #define BTRFS_EMPTY_DIR_SIZE 0
diff --git a/disk-io.c b/disk-io.c
index 72c672919cf9..59e297e2039c 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
 #include "print-tree.h"
 #include "common/rbtree-utils.h"
 #include "common/device-scan.h"
+#include "crypto/hash.h"
 
 /* specified errno for check_tree_block */
 #define BTRFS_BAD_BYTENR		(-1)
@@ -149,6 +150,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 		crc = crc32c(crc, data, len);
 		put_unaligned_le32(~crc, out);
 		return 0;
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return hash_xxhash(data, len, out);
 	default:
 		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
 		ASSERT(0);
diff --git a/mkfs/main.c b/mkfs/main.c
index 04509e788a52..da40e958fe0a 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -393,6 +393,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
 {
 	if (strcasecmp(s, "crc32c") == 0) {
 		return BTRFS_CSUM_TYPE_CRC32;
+	} else if (strcasecmp(s, "xxhash64") == 0 ||
+		   strcasecmp(s, "xxhash") == 0) {
+		return BTRFS_CSUM_TYPE_XXHASH;
 	} else {
 		error("unknown csum type %s", s);
 		exit(1);
-- 
2.16.4

Re: [PATCH v5 5/7] btrfs-progs: add xxhash64 to mkfs

[Johannes Thumshirn]

On 25/09/2019 15:37, Johannes Thumshirn wrote:
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  Makefile                  |  3 ++-
>  cmds/inspect-dump-super.c |  1 +
>  crypto/hash.c             | 17 +++++++++++++++++
>  crypto/hash.h             | 10 ++++++++++
>  ctree.h                   |  3 ++-
>  disk-io.c                 |  3 +++
>  mkfs/main.c               |  3 +++
>  7 files changed, 38 insertions(+), 2 deletions(-)
>  create mode 100644 crypto/hash.c
>  create mode 100644 crypto/hash.h
> 
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
>  	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
>  libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
>  		   kernel-lib/crc32c.o common/messages.o \
> -		   uuid-tree.o utils-lib.o common/rbtree-utils.o
> +		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
> +		   crypto/hash.o crypto/xxhash.o
>  libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
>  	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
>  	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index bf380ad2b56a..73e986ed8ee8 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -315,6 +315,7 @@ static bool is_valid_csum_type(u16 csum_type)
>  {
>  	switch (csum_type) {
>  	case BTRFS_CSUM_TYPE_CRC32:
> +	case BTRFS_CSUM_TYPE_XXHASH:
>  		return true;
>  	default:
>  		return false;
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..58a3890bd467
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,17 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> +	XXH64_hash_t hash;
> +
> +	hash = XXH64(buf, length, 0);
> +	/*
> +	 * NOTE: we're not taking the canonical form here but the plain hash to
> +	 * be compatible with the kernel implementation!
> +	 */
> +	memcpy(out, &hash, 8);
> +
> +	return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX	32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.h b/ctree.h
> index f70271dc658e..144c89eb4a36 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -166,7 +166,8 @@ struct btrfs_free_space_ctl;
>  
>  /* csum types */
>  enum btrfs_csum_type {
> -	BTRFS_CSUM_TYPE_CRC32   = 0,
> +	BTRFS_CSUM_TYPE_CRC32	= 0,
> +	BTRFS_CSUM_TYPE_XXHASH	= 1,
>  };
>  
>  #define BTRFS_EMPTY_DIR_SIZE 0
> diff --git a/disk-io.c b/disk-io.c
> index 72c672919cf9..59e297e2039c 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
>  #include "print-tree.h"
>  #include "common/rbtree-utils.h"
>  #include "common/device-scan.h"
> +#include "crypto/hash.h"
>  
>  /* specified errno for check_tree_block */
>  #define BTRFS_BAD_BYTENR		(-1)
> @@ -149,6 +150,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
>  		crc = crc32c(crc, data, len);
>  		put_unaligned_le32(~crc, out);
>  		return 0;
> +	case BTRFS_CSUM_TYPE_XXHASH:
> +		return hash_xxhash(data, len, out);
>  	default:
>  		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
>  		ASSERT(0);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 04509e788a52..da40e958fe0a 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -393,6 +393,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
>  {
>  	if (strcasecmp(s, "crc32c") == 0) {
>  		return BTRFS_CSUM_TYPE_CRC32;
> +	} else if (strcasecmp(s, "xxhash64") == 0 ||
> +		   strcasecmp(s, "xxhash") == 0) {
> +		return BTRFS_CSUM_TYPE_XXHASH;
>  	} else {
>  		error("unknown csum type %s", s);
>  		exit(1);
> 

This one lost this hunk in ctree.c:

diff --git a/ctree.c b/ctree.c
index 5ad291d0f4ac..91d685ff90a4 100644
--- a/ctree.c
+++ b/ctree.c
@@ -43,6 +43,7 @@ static const struct btrfs_csum {
        const char name[14];
 } btrfs_csums[] = {
        [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+       [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
 };

 u16 btrfs_super_csum_size(const struct btrfs_super_block *sb)

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH v5 5/7] btrfs-progs: add xxhash64 to mkfs

[David Sterba]

On Thu, Sep 26, 2019 at 12:06:53PM +0200, Johannes Thumshirn wrote:
> On 25/09/2019 15:37, Johannes Thumshirn wrote:
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  Makefile                  |  3 ++-
> >  cmds/inspect-dump-super.c |  1 +
> >  crypto/hash.c             | 17 +++++++++++++++++
> >  crypto/hash.h             | 10 ++++++++++
> >  ctree.h                   |  3 ++-
> >  disk-io.c                 |  3 +++
> >  mkfs/main.c               |  3 +++
> >  7 files changed, 38 insertions(+), 2 deletions(-)
> >  create mode 100644 crypto/hash.c
> >  create mode 100644 crypto/hash.h
> > 
> > diff --git a/Makefile b/Makefile
> > index 370e0c37ff65..45530749e2b9 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> >  	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
> >  libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> >  		   kernel-lib/crc32c.o common/messages.o \
> > -		   uuid-tree.o utils-lib.o common/rbtree-utils.o
> > +		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
> > +		   crypto/hash.o crypto/xxhash.o
> >  libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> >  	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> >  	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> > diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> > index bf380ad2b56a..73e986ed8ee8 100644
> > --- a/cmds/inspect-dump-super.c
> > +++ b/cmds/inspect-dump-super.c
> > @@ -315,6 +315,7 @@ static bool is_valid_csum_type(u16 csum_type)
> >  {
> >  	switch (csum_type) {
> >  	case BTRFS_CSUM_TYPE_CRC32:
> > +	case BTRFS_CSUM_TYPE_XXHASH:
> >  		return true;
> >  	default:
> >  		return false;
> > diff --git a/crypto/hash.c b/crypto/hash.c
> > new file mode 100644
> > index 000000000000..58a3890bd467
> > --- /dev/null
> > +++ b/crypto/hash.c
> > @@ -0,0 +1,17 @@
> > +#include "crypto/hash.h"
> > +#include "crypto/xxhash.h"
> > +
> > +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> > +{
> > +	XXH64_hash_t hash;
> > +
> > +	hash = XXH64(buf, length, 0);
> > +	/*
> > +	 * NOTE: we're not taking the canonical form here but the plain hash to
> > +	 * be compatible with the kernel implementation!
> > +	 */
> > +	memcpy(out, &hash, 8);
> > +
> > +	return 0;
> > +}
> > +
> > diff --git a/crypto/hash.h b/crypto/hash.h
> > new file mode 100644
> > index 000000000000..45c1ef17bc57
> > --- /dev/null
> > +++ b/crypto/hash.h
> > @@ -0,0 +1,10 @@
> > +#ifndef CRYPTO_HASH_H
> > +#define CRYPTO_HASH_H
> > +
> > +#include "../kerncompat.h"
> > +
> > +#define CRYPTO_HASH_SIZE_MAX	32
> > +
> > +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> > +
> > +#endif
> > diff --git a/ctree.h b/ctree.h
> > index f70271dc658e..144c89eb4a36 100644
> > --- a/ctree.h
> > +++ b/ctree.h
> > @@ -166,7 +166,8 @@ struct btrfs_free_space_ctl;
> >  
> >  /* csum types */
> >  enum btrfs_csum_type {
> > -	BTRFS_CSUM_TYPE_CRC32   = 0,
> > +	BTRFS_CSUM_TYPE_CRC32	= 0,
> > +	BTRFS_CSUM_TYPE_XXHASH	= 1,
> >  };
> >  
> >  #define BTRFS_EMPTY_DIR_SIZE 0
> > diff --git a/disk-io.c b/disk-io.c
> > index 72c672919cf9..59e297e2039c 100644
> > --- a/disk-io.c
> > +++ b/disk-io.c
> > @@ -34,6 +34,7 @@
> >  #include "print-tree.h"
> >  #include "common/rbtree-utils.h"
> >  #include "common/device-scan.h"
> > +#include "crypto/hash.h"
> >  
> >  /* specified errno for check_tree_block */
> >  #define BTRFS_BAD_BYTENR		(-1)
> > @@ -149,6 +150,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
> >  		crc = crc32c(crc, data, len);
> >  		put_unaligned_le32(~crc, out);
> >  		return 0;
> > +	case BTRFS_CSUM_TYPE_XXHASH:
> > +		return hash_xxhash(data, len, out);
> >  	default:
> >  		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
> >  		ASSERT(0);
> > diff --git a/mkfs/main.c b/mkfs/main.c
> > index 04509e788a52..da40e958fe0a 100644
> > --- a/mkfs/main.c
> > +++ b/mkfs/main.c
> > @@ -393,6 +393,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> >  {
> >  	if (strcasecmp(s, "crc32c") == 0) {
> >  		return BTRFS_CSUM_TYPE_CRC32;
> > +	} else if (strcasecmp(s, "xxhash64") == 0 ||
> > +		   strcasecmp(s, "xxhash") == 0) {
> > +		return BTRFS_CSUM_TYPE_XXHASH;
> >  	} else {
> >  		error("unknown csum type %s", s);
> >  		exit(1);
> > 
> 
> This one lost this hunk in ctree.c:
> 
> diff --git a/ctree.c b/ctree.c
> index 5ad291d0f4ac..91d685ff90a4 100644
> --- a/ctree.c
> +++ b/ctree.c
> @@ -43,6 +43,7 @@ static const struct btrfs_csum {
>         const char name[14];
>  } btrfs_csums[] = {
>         [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> +       [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
>  };
> 
>  u16 btrfs_super_csum_size(const struct btrfs_super_block *sb)

Folded to the patch and lightly tested. Thanks.

[PATCH v6] btrfs-progs: add xxhash64 to mkfs

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---

Changes since v5:
- add xxhash64 to hash table

 Makefile                  |  3 ++-
 cmds/inspect-dump-super.c |  1 +
 crypto/hash.c             | 16 ++++++++++++++++
 crypto/hash.h             | 10 ++++++++++
 ctree.c                   |  1 +
 ctree.h                   |  3 ++-
 disk-io.c                 |  3 +++
 mkfs/main.c               |  3 +++
 8 files changed, 38 insertions(+), 2 deletions(-)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h

diff --git a/Makefile b/Makefile
index 370e0c37ff65..45530749e2b9 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
 	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
 		   kernel-lib/crc32c.o common/messages.o \
-		   uuid-tree.o utils-lib.o common/rbtree-utils.o
+		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
+		   crypto/hash.o crypto/xxhash.o
 libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
 	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
 	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index bf380ad2b56a..73e986ed8ee8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -315,6 +315,7 @@ static bool is_valid_csum_type(u16 csum_type)
 {
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
 		return true;
 	default:
 		return false;
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..8c428cba11f0
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,16 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+	XXH64_hash_t hash;
+
+	hash = XXH64(buf, length, 0);
+	/*
+	 * NOTE: we're not taking the canonical form here but the plain hash to
+	 * be compatible with the kernel implementation!
+	 */
+	memcpy(out, &hash, 8);
+
+	return 0;
+}
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX	32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/ctree.c b/ctree.c
index a52ccfe19f94..139ffd613da5 100644
--- a/ctree.c
+++ b/ctree.c
@@ -43,6 +43,7 @@ static struct btrfs_csum {
 	const char *name;
 } btrfs_csums[] = {
 	[BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+	[BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
 };
 
 u16 btrfs_super_csum_size(const struct btrfs_super_block *sb)
diff --git a/ctree.h b/ctree.h
index f70271dc658e..144c89eb4a36 100644
--- a/ctree.h
+++ b/ctree.h
@@ -166,7 +166,8 @@ struct btrfs_free_space_ctl;
 
 /* csum types */
 enum btrfs_csum_type {
-	BTRFS_CSUM_TYPE_CRC32   = 0,
+	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
 };
 
 #define BTRFS_EMPTY_DIR_SIZE 0
diff --git a/disk-io.c b/disk-io.c
index 72c672919cf9..59e297e2039c 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
 #include "print-tree.h"
 #include "common/rbtree-utils.h"
 #include "common/device-scan.h"
+#include "crypto/hash.h"
 
 /* specified errno for check_tree_block */
 #define BTRFS_BAD_BYTENR		(-1)
@@ -149,6 +150,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 		crc = crc32c(crc, data, len);
 		put_unaligned_le32(~crc, out);
 		return 0;
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return hash_xxhash(data, len, out);
 	default:
 		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
 		ASSERT(0);
diff --git a/mkfs/main.c b/mkfs/main.c
index f52e8b61a460..a6deddc47c69 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -392,6 +392,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
 {
 	if (strcasecmp(s, "crc32c") == 0) {
 		return BTRFS_CSUM_TYPE_CRC32;
+	} else if (strcasecmp(s, "xxhash64") == 0 ||
+		   strcasecmp(s, "xxhash") == 0) {
+		return BTRFS_CSUM_TYPE_XXHASH;
 	} else {
 		error("unknown csum type %s", s);
 		exit(1);
-- 
2.16.4

Re: [PATCH v6] btrfs-progs: add xxhash64 to mkfs

[Qu Wenruo]

On 2019/9/26 下午6:11, Johannes Thumshirn wrote:
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---

Not related to the patchset itself, but it would be pretty nice if we
check the sysfs interface to guess if we can mount the fs.

And if not supported, a warning (at stdout) will not hurt.

Thanks,
Qu

>
> Changes since v5:
> - add xxhash64 to hash table
>
>  Makefile                  |  3 ++-
>  cmds/inspect-dump-super.c |  1 +
>  crypto/hash.c             | 16 ++++++++++++++++
>  crypto/hash.h             | 10 ++++++++++
>  ctree.c                   |  1 +
>  ctree.h                   |  3 ++-
>  disk-io.c                 |  3 +++
>  mkfs/main.c               |  3 +++
>  8 files changed, 38 insertions(+), 2 deletions(-)
>  create mode 100644 crypto/hash.c
>  create mode 100644 crypto/hash.h
>
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
>  	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
>  libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
>  		   kernel-lib/crc32c.o common/messages.o \
> -		   uuid-tree.o utils-lib.o common/rbtree-utils.o
> +		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
> +		   crypto/hash.o crypto/xxhash.o
>  libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
>  	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
>  	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index bf380ad2b56a..73e986ed8ee8 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -315,6 +315,7 @@ static bool is_valid_csum_type(u16 csum_type)
>  {
>  	switch (csum_type) {
>  	case BTRFS_CSUM_TYPE_CRC32:
> +	case BTRFS_CSUM_TYPE_XXHASH:
>  		return true;
>  	default:
>  		return false;
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..8c428cba11f0
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> +	XXH64_hash_t hash;
> +
> +	hash = XXH64(buf, length, 0);
> +	/*
> +	 * NOTE: we're not taking the canonical form here but the plain hash to
> +	 * be compatible with the kernel implementation!
> +	 */
> +	memcpy(out, &hash, 8);
> +
> +	return 0;
> +}
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX	32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.c b/ctree.c
> index a52ccfe19f94..139ffd613da5 100644
> --- a/ctree.c
> +++ b/ctree.c
> @@ -43,6 +43,7 @@ static struct btrfs_csum {
>  	const char *name;
>  } btrfs_csums[] = {
>  	[BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> +	[BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
>  };
>
>  u16 btrfs_super_csum_size(const struct btrfs_super_block *sb)
> diff --git a/ctree.h b/ctree.h
> index f70271dc658e..144c89eb4a36 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -166,7 +166,8 @@ struct btrfs_free_space_ctl;
>
>  /* csum types */
>  enum btrfs_csum_type {
> -	BTRFS_CSUM_TYPE_CRC32   = 0,
> +	BTRFS_CSUM_TYPE_CRC32	= 0,
> +	BTRFS_CSUM_TYPE_XXHASH	= 1,
>  };
>
>  #define BTRFS_EMPTY_DIR_SIZE 0
> diff --git a/disk-io.c b/disk-io.c
> index 72c672919cf9..59e297e2039c 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
>  #include "print-tree.h"
>  #include "common/rbtree-utils.h"
>  #include "common/device-scan.h"
> +#include "crypto/hash.h"
>
>  /* specified errno for check_tree_block */
>  #define BTRFS_BAD_BYTENR		(-1)
> @@ -149,6 +150,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
>  		crc = crc32c(crc, data, len);
>  		put_unaligned_le32(~crc, out);
>  		return 0;
> +	case BTRFS_CSUM_TYPE_XXHASH:
> +		return hash_xxhash(data, len, out);
>  	default:
>  		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
>  		ASSERT(0);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index f52e8b61a460..a6deddc47c69 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -392,6 +392,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
>  {
>  	if (strcasecmp(s, "crc32c") == 0) {
>  		return BTRFS_CSUM_TYPE_CRC32;
> +	} else if (strcasecmp(s, "xxhash64") == 0 ||
> +		   strcasecmp(s, "xxhash") == 0) {
> +		return BTRFS_CSUM_TYPE_XXHASH;
>  	} else {
>  		error("unknown csum type %s", s);
>  		exit(1);
>

Re: [PATCH v6] btrfs-progs: add xxhash64 to mkfs

[David Sterba]

On Mon, Oct 07, 2019 at 07:11:17PM +0800, Qu Wenruo wrote:
> On 2019/9/26 下午6:11, Johannes Thumshirn wrote:
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> 
> Not related to the patchset itself, but it would be pretty nice if we
> check the sysfs interface to guess if we can mount the fs.
> 
> And if not supported, a warning (at stdout) will not hurt.

We don't do that with any other feature that's selected at mkfs time and
not supported by the running kernel so this would have to be done more
thoroughly, not just for checksums.

[PATCH v5 6/7] btrfs-progs: move crc32c implementation to crypto/

[Johannes Thumshirn]

With the introduction of xxhash64 to btrfs-progs we created a crypto/
directory for all the hashes used in btrfs (although no
cryptographically secure hash is there yet).

Move the crc32c implementation from kernel-lib/ to crypto/ as well so we
have all hashes consolidated.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 Android.mk                      | 4 ++--
 Makefile                        | 4 ++--
 btrfs-crc.c                     | 2 +-
 btrfs-find-root.c               | 2 +-
 btrfs-sb-mod.c                  | 2 +-
 btrfs.c                         | 2 +-
 cmds/inspect-dump-super.c       | 2 +-
 cmds/rescue-chunk-recover.c     | 2 +-
 cmds/rescue-super-recover.c     | 2 +-
 common/utils.c                  | 2 +-
 convert/main.c                  | 2 +-
 {kernel-lib => crypto}/crc32c.c | 2 +-
 {kernel-lib => crypto}/crc32c.h | 0
 disk-io.c                       | 2 +-
 extent-tree.c                   | 2 +-
 file-item.c                     | 2 +-
 free-space-cache.c              | 2 +-
 hash.h                          | 2 +-
 image/main.c                    | 2 +-
 image/sanitize.c                | 2 +-
 library-test.c                  | 2 +-
 mkfs/main.c                     | 2 +-
 send-stream.c                   | 2 +-
 23 files changed, 24 insertions(+), 24 deletions(-)
 rename {kernel-lib => crypto}/crc32c.c (99%)
 rename {kernel-lib => crypto}/crc32c.h (100%)

diff --git a/Android.mk b/Android.mk
index e8de47eb4617..8288ba7356f4 100644
--- a/Android.mk
+++ b/Android.mk
@@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \
                cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \
                mkfs/common.c
 libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \
-                   kernel-lib/crc32c.c messages.c \
+                   crypto/crc32c.c messages.c \
                    uuid-tree.c utils-lib.c rbtree-utils.c
 libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
-                   kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+                   crypto/crc32c.h kernel-lib/list.h kerncompat.h \
                    kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
                    extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
 blkid_objects := partition/ superblocks/ topology/
diff --git a/Makefile b/Makefile
index 45530749e2b9..c241c22d1018 100644
--- a/Makefile
+++ b/Makefile
@@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
 	       cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \
 	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
-		   kernel-lib/crc32c.o common/messages.o \
+		   crypto/crc32c.o common/messages.o \
 		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
 		   crypto/hash.o crypto/xxhash.o
 libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
-	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+	       crypto/crc32c.h kernel-lib/list.h kerncompat.h \
 	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
 	       extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
 libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h)
diff --git a/btrfs-crc.c b/btrfs-crc.c
index bcf25df8b46a..c4f81fc65f67 100644
--- a/btrfs-crc.c
+++ b/btrfs-crc.c
@@ -19,7 +19,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/utils.h"
 
 void print_usage(int status)
diff --git a/btrfs-find-root.c b/btrfs-find-root.c
index e46fa8723b33..741eb9a95ac5 100644
--- a/btrfs-find-root.c
+++ b/btrfs-find-root.c
@@ -32,7 +32,7 @@
 #include "kernel-lib/list.h"
 #include "volumes.h"
 #include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "extent-cache.h"
 #include "find-root.h"
 #include "common/help.h"
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 105b556b0cf1..348991b39451 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -24,7 +24,7 @@
 #include <string.h>
 #include <limits.h>
 #include <byteswap.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "disk-io.h"
 
 #define BLOCKSIZE (4096)
diff --git a/btrfs.c b/btrfs.c
index 6c8aabe24dc8..72dad6fb3983 100644
--- a/btrfs.c
+++ b/btrfs.c
@@ -20,7 +20,7 @@
 #include <getopt.h>
 
 #include "volumes.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "cmds/commands.h"
 #include "common/utils.h"
 #include "common/help.h"
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 73e986ed8ee8..52636bd2e11c 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -32,7 +32,7 @@
 #include "kernel-lib/list.h"
 #include "common/utils.h"
 #include "cmds/commands.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/help.h"
 
 static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 329a608dfc6b..bf35693ddbfa 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -36,7 +36,7 @@
 #include "disk-io.h"
 #include "volumes.h"
 #include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/utils.h"
 #include "check/common.h"
 #include "cmds/commands.h"
diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c
index ea3a00caf56c..5d6bea836c8b 100644
--- a/cmds/rescue-super-recover.c
+++ b/cmds/rescue-super-recover.c
@@ -31,7 +31,7 @@
 #include "disk-io.h"
 #include "kernel-lib/list.h"
 #include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "volumes.h"
 #include "cmds/commands.h"
 #include "cmds/rescue.h"
diff --git a/common/utils.c b/common/utils.c
index f2a10cccca86..fa49c01ad102 100644
--- a/common/utils.c
+++ b/common/utils.c
@@ -47,7 +47,7 @@
 #include "ctree.h"
 #include "disk-io.h"
 #include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/utils.h"
 #include "common/path-utils.h"
 #include "common/device-scan.h"
diff --git a/convert/main.c b/convert/main.c
index bb689be9f3e4..3686b5814838 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -102,7 +102,7 @@
 #include "mkfs/common.h"
 #include "convert/common.h"
 #include "convert/source-fs.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/fsfeatures.h"
 #include "common/box.h"
 
diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c
similarity index 99%
rename from kernel-lib/crc32c.c
rename to crypto/crc32c.c
index 36bb6f189971..bd6283d5baeb 100644
--- a/kernel-lib/crc32c.c
+++ b/crypto/crc32c.c
@@ -8,7 +8,7 @@
  *
  */
 #include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include <inttypes.h>
 #include <string.h>
 #include <unistd.h>
diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h
similarity index 100%
rename from kernel-lib/crc32c.h
rename to crypto/crc32c.h
diff --git a/disk-io.c b/disk-io.c
index 59e297e2039c..8f15adbf4910 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -29,7 +29,7 @@
 #include "disk-io.h"
 #include "volumes.h"
 #include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/utils.h"
 #include "print-tree.h"
 #include "common/rbtree-utils.h"
diff --git a/extent-tree.c b/extent-tree.c
index 932af2c644bd..a8f57776bd73 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -26,7 +26,7 @@
 #include "disk-io.h"
 #include "print-tree.h"
 #include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "volumes.h"
 #include "free-space-cache.h"
 #include "free-space-tree.h"
diff --git a/file-item.c b/file-item.c
index c6e9d212bcab..64af57693baf 100644
--- a/file-item.c
+++ b/file-item.c
@@ -24,7 +24,7 @@
 #include "disk-io.h"
 #include "transaction.h"
 #include "print-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/internal.h"
 
 #define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
diff --git a/free-space-cache.c b/free-space-cache.c
index 8a57f86dc650..6e7d7e1ef561 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -22,7 +22,7 @@
 #include "transaction.h"
 #include "disk-io.h"
 #include "extent_io.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "kernel-lib/bitops.h"
 #include "common/internal.h"
 #include "common/utils.h"
diff --git a/hash.h b/hash.h
index 5398e8869316..51e842047093 100644
--- a/hash.h
+++ b/hash.h
@@ -19,7 +19,7 @@
 #ifndef __BTRFS_HASH_H__
 #define __BTRFS_HASH_H__
 
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 
 static inline u64 btrfs_name_hash(const char *name, int len)
 {
diff --git a/image/main.c b/image/main.c
index f713e47b85cb..b87525412543 100644
--- a/image/main.c
+++ b/image/main.c
@@ -28,7 +28,7 @@
 #include <getopt.h>
 
 #include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "ctree.h"
 #include "disk-io.h"
 #include "transaction.h"
diff --git a/image/sanitize.c b/image/sanitize.c
index 9caa5deaf2dd..cd2bd6fe2379 100644
--- a/image/sanitize.c
+++ b/image/sanitize.c
@@ -18,7 +18,7 @@
 #include "common/internal.h"
 #include "common/messages.h"
 #include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "image/sanitize.h"
 #include "extent_io.h"
 
diff --git a/library-test.c b/library-test.c
index c44bad228e50..e47917c25830 100644
--- a/library-test.c
+++ b/library-test.c
@@ -21,7 +21,7 @@
 #include "version.h"
 #include "kernel-lib/rbtree.h"
 #include "kernel-lib/radix-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "kernel-lib/list.h"
 #include "kernel-lib/sizes.h"
 #include "ctree.h"
diff --git a/mkfs/main.c b/mkfs/main.c
index da40e958fe0a..a0dc5846beb3 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -45,7 +45,7 @@
 #include "common/rbtree-utils.h"
 #include "mkfs/common.h"
 #include "mkfs/rootdir.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/fsfeatures.h"
 #include "common/box.h"
 
diff --git a/send-stream.c b/send-stream.c
index 08687e5e6904..3b8ada2110aa 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -21,7 +21,7 @@
 
 #include "send.h"
 #include "send-stream.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
 #include "common/utils.h"
 
 struct btrfs_send_stream {
-- 
2.16.4

[RFC PATCH v5 7/7] btrfs-progs: add test override for mkfs to use different checksums

[Johannes Thumshirn]

Similar to check's test overrides add an override for mkfs tests so we can
specify different mkfs flags.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 tests/common                                | 10 ++++++++--
 tests/mkfs-tests/001-basic-profiles/test.sh |  8 +++++++-
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/tests/common b/tests/common
index 75e5540155cc..5148820bef58 100644
--- a/tests/common
+++ b/tests/common
@@ -473,16 +473,22 @@ prepare_test_dev()
 # $1-$n: optional, default is -f
 run_check_mkfs_test_dev()
 {
+	MKFS_ARGS="$@"
+
 	setup_root_helper
 
 	# check accidental files/devices passed
-	for opt in "$@"; do
+	for opt in "$MKFS_ARGS"; do
 		if [ -f "$opt" -o -b "$opt" ]; then
 			_fail "ERROR: unexpected option for run_check_mkfs_test_dev: device"
 		fi
 	done
 
-	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$@" "$TEST_DEV"
+	if [ "$TEST_ENABLE_OVERRIDE" = 'true' ]; then
+		MKFS_ARGS="$TEST_ARGS_MKFS $MKFS_ARGS"
+	fi
+
+	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$MKFS_ARGS" "$TEST_DEV"
 }
 
 run_check_mount_test_dev()
diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 6e295274119d..e0110c722555 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -21,7 +21,13 @@ test_get_info()
 }
 test_do_mkfs()
 {
-	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$@"
+	MKFS_ARGS="$@"
+
+	if [ "$TEST_ENABLE_OVERRIDE" = 'true' ]; then
+		MKFS_ARGS="$TEST_ARGS_MKFS $MKFS_ARGS"
+	fi
+
+	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$MKFS_ARGS"
 }
 
 test_mkfs_single()
-- 
2.16.4

Re: [RFC PATCH v5 7/7] btrfs-progs: add test override for mkfs to use different checksums

[David Sterba]

On Wed, Sep 25, 2019 at 03:37:28PM +0200, Johannes Thumshirn wrote:
> Similar to check's test overrides add an override for mkfs tests so we can
> specify different mkfs flags.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  tests/common                                | 10 ++++++++--
>  tests/mkfs-tests/001-basic-profiles/test.sh |  8 +++++++-
>  2 files changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/common b/tests/common
> index 75e5540155cc..5148820bef58 100644
> --- a/tests/common
> +++ b/tests/common
> @@ -473,16 +473,22 @@ prepare_test_dev()
>  # $1-$n: optional, default is -f
>  run_check_mkfs_test_dev()
>  {
> +	MKFS_ARGS="$@"
> +
>  	setup_root_helper
>  
>  	# check accidental files/devices passed
> -	for opt in "$@"; do
> +	for opt in "$MKFS_ARGS"; do
>  		if [ -f "$opt" -o -b "$opt" ]; then
>  			_fail "ERROR: unexpected option for run_check_mkfs_test_dev: device"
>  		fi
>  	done
>  
> -	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$@" "$TEST_DEV"
> +	if [ "$TEST_ENABLE_OVERRIDE" = 'true' ]; then
> +		MKFS_ARGS="$TEST_ARGS_MKFS $MKFS_ARGS"
> +	fi
> +
> +	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$MKFS_ARGS" "$TEST_DEV"
>  }
>  
>  run_check_mount_test_dev()
> diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
> index 6e295274119d..e0110c722555 100755
> --- a/tests/mkfs-tests/001-basic-profiles/test.sh
> +++ b/tests/mkfs-tests/001-basic-profiles/test.sh
> @@ -21,7 +21,13 @@ test_get_info()
>  }
>  test_do_mkfs()
>  {
> -	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$@"
> +	MKFS_ARGS="$@"
> +
> +	if [ "$TEST_ENABLE_OVERRIDE" = 'true' ]; then
> +		MKFS_ARGS="$TEST_ARGS_MKFS $MKFS_ARGS"
> +	fi
> +
> +	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$MKFS_ARGS"

This is supposed to be transparent for all tests, so all the override
happens inside tests/common in _cmd_spec, so all what's needed is to add
a line to the 'case'.

Re: [PATCH v5 0/7] btrfs-progs: support xxhash64 checksums

[David Sterba]

On Wed, Sep 25, 2019 at 03:37:21PM +0200, Johannes Thumshirn wrote:
> Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
> prepared for an easy addition of new checksums, this patchset implements
> XXHASH64 as a second, fast but not cryptographically secure checksum hash.
> 
> This patchset is fully bisectible and available on github at
> https://github.com/morbidrsa/btrfs-progs/tree/mkfs-xxhash64.v5
> 
> Changes since v4:
> - Rebased onto latest 'devel' branch and dropped applied changes
> - Split 'btrfs-progs: add xxhash64 as checksum algorithm' into several atomic
>   patches
> - Changed test code to using 'TEST_ENABLE_OVERRIDE'
> 
> Johannes Thumshirn (7):
>   btrfs-progs: add option for checksum type to mkfs
>   btrfs-progs: add is_valid_csum_type() helper
>   btrfs-progs: add table for checksum type and name
>   btrfs-progs: also print checksum type when running mkfs
>   btrfs-progs: add xxhash64 to mkfs
>   btrfs-progs: move crc32c implementation to crypto/

All of the above added to devel, thanks. There might be some fixups or
cleanups but I'll do them as separate patches. Regarding relese, we
could do the xxhash support in 5.3, though there's no kernel support
yet. I'll think about that if this would not cause too much confusion
though.

Re: [PATCH v5 0/7] btrfs-progs: support xxhash64 checksums

[Johannes Thumshirn]

On 25/09/2019 19:22, David Sterba wrote:
[...]
> All of the above added to devel, thanks. There might be some fixups or
> cleanups but I'll do them as separate patches. Regarding relese, we
> could do the xxhash support in 5.3, though there's no kernel support
> yet. I'll think about that if this would not cause too much confusion
> though.

Thanks! I was about to re-send the kernel side (maybe w/o sysfs) today if
you don't mind.

Byte,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850