[RFC PATCH 0/4] Support xxhash64 checksums

[RFC PATCH 0/4] Support xxhash64 checksums

[Johannes Thumshirn]

Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
prepared for an easy addition of new checksums, this patchset implements
XXHASH64 as a second, fast but not cryptographically secure checksum hash.

This still is in RFC state (especially the btrfs-progs side).

David Sterba (1):
  btrfs: sysfs: export supported checksums

Johannes Thumshirn (3):
  btrfs: turn checksum type define into a union
  btrfs: create structure to encode checksum type and length
  btrfs: use xxhash64 for checksumming

 fs/btrfs/Kconfig                |  1 +
 fs/btrfs/ctree.h                | 17 ++++++++++++-----
 fs/btrfs/disk-io.c              |  1 +
 fs/btrfs/super.c                |  1 +
 fs/btrfs/sysfs.c                | 35 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/btrfs_tree.h |  5 ++++-
 6 files changed, 54 insertions(+), 6 deletions(-)

-- 
2.16.4

[RFC PATCH 1/4] btrfs: turn checksum type define into a union

[Johannes Thumshirn]

Turn the checksum type definition into a union. This eases later addition
of new checksums.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 include/uapi/linux/btrfs_tree.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index 34d5b34286fa..babef98181a2 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -300,7 +300,9 @@
 #define BTRFS_CSUM_SIZE 32
 
 /* csum types */
-#define BTRFS_CSUM_TYPE_CRC32	0
+enum btrfs_csum_type {
+	BTRFS_CSUM_TYPE_CRC32	= 0,
+};
 
 /*
  * flags definitions for directory entry item type
-- 
2.16.4

Re: [RFC PATCH 1/4] btrfs: turn checksum type define into a union

[Mike Fleetwood]

On Thu, 25 Jul 2019 at 10:34, Johannes Thumshirn <jthumshirn@suse.de> wrote:
>
> Turn the checksum type definition into a union. This eases later addition
> of new checksums.

I think you meant to say "Turn the checksum type definition into an
_enum_." in the title and commit message.


>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  include/uapi/linux/btrfs_tree.h | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
> index 34d5b34286fa..babef98181a2 100644
> --- a/include/uapi/linux/btrfs_tree.h
> +++ b/include/uapi/linux/btrfs_tree.h
> @@ -300,7 +300,9 @@
>  #define BTRFS_CSUM_SIZE 32
>
>  /* csum types */
> -#define BTRFS_CSUM_TYPE_CRC32  0
> +enum btrfs_csum_type {
> +       BTRFS_CSUM_TYPE_CRC32   = 0,
> +};
>
>  /*
>   * flags definitions for directory entry item type
> --
> 2.16.4
>

Re: [RFC PATCH 1/4] btrfs: turn checksum type define into a union

[Johannes Thumshirn]

On Thu, Jul 25, 2019 at 12:08:44PM +0100, Mike Fleetwood wrote:
> On Thu, 25 Jul 2019 at 10:34, Johannes Thumshirn <jthumshirn@suse.de> wrote:
> >
> > Turn the checksum type definition into a union. This eases later addition
> > of new checksums.
> 
> I think you meant to say "Turn the checksum type definition into an
> _enum_." in the title and commit message.

Indeed I did want to write 'enum'. No idea why I wrote union o.O

Thanks for spotting,
	JOhannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[RFC PATCH 2/4] btrfs: create structure to encode checksum type and length

[Johannes Thumshirn]

Create a structure to encode the type and length for the known on-disk
checksums. Also add a table and a convenience macro for adding the
checksum types to the table.

This makes it easier to add new checksums later.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/ctree.h | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index da97ff10f421..099401f5dd47 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -82,9 +82,15 @@ struct btrfs_ref;
  */
 #define BTRFS_LINK_MAX 65535U
 
-/* four bytes for CRC32 */
-static const int btrfs_csum_sizes[] = { 4 };
-static const char *btrfs_csum_names[] = { "crc32c" };
+#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
+	[_type] = { .size = _size, .name = _name }
+
+static const struct btrfs_csums {
+	u16		size;
+	const char	*name;
+} btrfs_csums[] = {
+	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
+};
 
 #define BTRFS_EMPTY_DIR_SIZE 0
 
@@ -2373,13 +2379,13 @@ static inline int btrfs_super_csum_size(const struct btrfs_super_block *s)
 	/*
 	 * csum type is validated at mount time
 	 */
-	return btrfs_csum_sizes[t];
+	return btrfs_csums[t].size;
 }
 
 static inline const char *btrfs_super_csum_name(u16 csum_type)
 {
 	/* csum type is validated at mount time */
-	return btrfs_csum_names[csum_type];
+	return btrfs_csums[csum_type].name;
 }
 
 /*
-- 
2.16.4

Re: [RFC PATCH 2/4] btrfs: create structure to encode checksum type and length

[Su Yue]

On 2019/7/25 5:33 PM, Johannes Thumshirn wrote:
> Create a structure to encode the type and length for the known on-disk
> checksums. Also add a table and a convenience macro for adding the
> checksum types to the table.
>
> This makes it easier to add new checksums later.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>   fs/btrfs/ctree.h | 16 +++++++++++-----
>   1 file changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index da97ff10f421..099401f5dd47 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -82,9 +82,15 @@ struct btrfs_ref;
>    */
>   #define BTRFS_LINK_MAX 65535U
>
> -/* four bytes for CRC32 */
> -static const int btrfs_csum_sizes[] = { 4 };
> -static const char *btrfs_csum_names[] = { "crc32c" };
> +#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
> +	[_type] = { .size = _size, .name = _name }
> +
> +static const struct btrfs_csums {
> +	u16		size;
> +	const char	*name;
> +} btrfs_csums[] = {
> +	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
> +};
>
How about:

struct btrfs_csum {
         u16             size;
         const char      *name;
};

static const struct btrfs_csum btrfs_csums[] = {
#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
              [_type] = { .size = _size, .name = _name }

         BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
};

Since the macro BTRFS_CHECKSUM_TYPE is only used in array btrfs_csum
btrfs_csums. And this makes the struct btrfs_csum clear.


---
Su

>   #define BTRFS_EMPTY_DIR_SIZE 0
>
> @@ -2373,13 +2379,13 @@ static inline int btrfs_super_csum_size(const struct btrfs_super_block *s)
>   	/*
>   	 * csum type is validated at mount time
>   	 */
> -	return btrfs_csum_sizes[t];
> +	return btrfs_csums[t].size;
>   }
>
>   static inline const char *btrfs_super_csum_name(u16 csum_type)
>   {
>   	/* csum type is validated at mount time */
> -	return btrfs_csum_names[csum_type];
> +	return btrfs_csums[csum_type].name;
>   }
>
>   /*
>

Re: [RFC PATCH 2/4] btrfs: create structure to encode checksum type and length

[David Sterba]

On Thu, Jul 25, 2019 at 11:33:49AM +0200, Johannes Thumshirn wrote:
> Create a structure to encode the type and length for the known on-disk
> checksums. Also add a table and a convenience macro for adding the
> checksum types to the table.
> 
> This makes it easier to add new checksums later.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/ctree.h | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index da97ff10f421..099401f5dd47 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -82,9 +82,15 @@ struct btrfs_ref;
>   */
>  #define BTRFS_LINK_MAX 65535U
>  
> -/* four bytes for CRC32 */
> -static const int btrfs_csum_sizes[] = { 4 };
> -static const char *btrfs_csum_names[] = { "crc32c" };
> +#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
> +	[_type] = { .size = _size, .name = _name }

I think the macro initializer might be an overkill here, there are only
3 items to initialize.

> +
> +static const struct btrfs_csums {
> +	u16		size;
> +	const char	*name;
> +} btrfs_csums[] = {
> +	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
> +};
>  
>  #define BTRFS_EMPTY_DIR_SIZE 0
>  
> @@ -2373,13 +2379,13 @@ static inline int btrfs_super_csum_size(const struct btrfs_super_block *s)
>  	/*
>  	 * csum type is validated at mount time
>  	 */
> -	return btrfs_csum_sizes[t];
> +	return btrfs_csums[t].size;
>  }
>  
>  static inline const char *btrfs_super_csum_name(u16 csum_type)
>  {
>  	/* csum type is validated at mount time */
> -	return btrfs_csum_names[csum_type];
> +	return btrfs_csums[csum_type].name;
>  }

This has been in the code already, but shouldn't the btrfs_csums table
be declared in ctree.h and defined in eg. in ctree.c? As ctree.h is
included by everything we have multiple copies of it in the .o files.

Re: [RFC PATCH 2/4] btrfs: create structure to encode checksum type and length

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> Create a structure to encode the type and length for the known on-disk
> checksums. Also add a table and a convenience macro for adding the
> checksum types to the table.
> 
> This makes it easier to add new checksums later.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/ctree.h | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index da97ff10f421..099401f5dd47 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -82,9 +82,15 @@ struct btrfs_ref;
>   */
>  #define BTRFS_LINK_MAX 65535U
>  
> -/* four bytes for CRC32 */
> -static const int btrfs_csum_sizes[] = { 4 };
> -static const char *btrfs_csum_names[] = { "crc32c" };
> +#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
> +	[_type] = { .size = _size, .name = _name }
> +
> +static const struct btrfs_csums {
> +	u16		size;
> +	const char	*name;
> +} btrfs_csums[] = {
> +	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
> +};


Considering we won't support more than 4-5 csums  I'd rather you remove
the macro.

>  
>  #define BTRFS_EMPTY_DIR_SIZE 0
>  
> @@ -2373,13 +2379,13 @@ static inline int btrfs_super_csum_size(const struct btrfs_super_block *s)
>  	/*
>  	 * csum type is validated at mount time
>  	 */
> -	return btrfs_csum_sizes[t];
> +	return btrfs_csums[t].size;
>  }
>  
>  static inline const char *btrfs_super_csum_name(u16 csum_type)
>  {
>  	/* csum type is validated at mount time */
> -	return btrfs_csum_names[csum_type];
> +	return btrfs_csums[csum_type].name;
>  }
>  
>  /*
> 

Re: [RFC PATCH 2/4] btrfs: create structure to encode checksum type and length

[Johannes Thumshirn]

On Mon, Aug 12, 2019 at 12:07:44PM +0300, Nikolay Borisov wrote:
> 
> 
> On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> > Create a structure to encode the type and length for the known on-disk
> > checksums. Also add a table and a convenience macro for adding the
> > checksum types to the table.
> > 
> > This makes it easier to add new checksums later.
> > 
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  fs/btrfs/ctree.h | 16 +++++++++++-----
> >  1 file changed, 11 insertions(+), 5 deletions(-)
> > 
> > diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> > index da97ff10f421..099401f5dd47 100644
> > --- a/fs/btrfs/ctree.h
> > +++ b/fs/btrfs/ctree.h
> > @@ -82,9 +82,15 @@ struct btrfs_ref;
> >   */
> >  #define BTRFS_LINK_MAX 65535U
> >  
> > -/* four bytes for CRC32 */
> > -static const int btrfs_csum_sizes[] = { 4 };
> > -static const char *btrfs_csum_names[] = { "crc32c" };
> > +#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \
> > +	[_type] = { .size = _size, .name = _name }
> > +
> > +static const struct btrfs_csums {
> > +	u16		size;
> > +	const char	*name;
> > +} btrfs_csums[] = {
> > +	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
> > +};
> 
> 
> Considering we won't support more than 4-5 csums  I'd rather you remove
> the macro.

Yeah already dropped it, though I thought it might help a bit on the
readability/self-describing side.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[RFC PATCH 3/4] btrfs: use xxhash64 for checksumming

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/Kconfig                | 1 +
 fs/btrfs/ctree.h                | 1 +
 fs/btrfs/disk-io.c              | 1 +
 fs/btrfs/super.c                | 1 +
 include/uapi/linux/btrfs_tree.h | 1 +
 5 files changed, 5 insertions(+)

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 38651fae7f21..6d5a01c57da3 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -5,6 +5,7 @@ config BTRFS_FS
 	select CRYPTO
 	select CRYPTO_CRC32C
 	select LIBCRC32C
+	select CRYPTO_XXHASH
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 099401f5dd47..b34f22e55304 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -90,6 +90,7 @@ static const struct btrfs_csums {
 	const char	*name;
 } btrfs_csums[] = {
 	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
+	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_XXHASH, 8, "xxhash64"),
 };
 
 #define BTRFS_EMPTY_DIR_SIZE 0
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 5f7ee70b3d1a..54a8ef489850 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -351,6 +351,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
 {
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
 		return true;
 	default:
 		return false;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 78de9d5d80c6..7312f675d702 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2464,3 +2464,4 @@ module_exit(exit_btrfs_fs)
 
 MODULE_LICENSE("GPL");
 MODULE_SOFTDEP("pre: crc32c");
+MODULE_SOFTDEP("pre: xxhash64");
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index babef98181a2..af4f5dec10b7 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -302,6 +302,7 @@
 /* csum types */
 enum btrfs_csum_type {
 	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
 };
 
 /*
-- 
2.16.4

Re: [RFC PATCH 3/4] btrfs: use xxhash64 for checksumming

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2407 bytes --]



On 2019/7/25 下午5:33, Johannes Thumshirn wrote:
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/Kconfig                | 1 +
>  fs/btrfs/ctree.h                | 1 +
>  fs/btrfs/disk-io.c              | 1 +
>  fs/btrfs/super.c                | 1 +
>  include/uapi/linux/btrfs_tree.h | 1 +
>  5 files changed, 5 insertions(+)
> 
> diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
> index 38651fae7f21..6d5a01c57da3 100644
> --- a/fs/btrfs/Kconfig
> +++ b/fs/btrfs/Kconfig
> @@ -5,6 +5,7 @@ config BTRFS_FS
>  	select CRYPTO
>  	select CRYPTO_CRC32C
>  	select LIBCRC32C
> +	select CRYPTO_XXHASH

Just an off topic idea, can we make such CRYPTO_* support configurable?
E.g. make something like CONFIG_BTRFS_CRYPTO_XXHASH.

Not sure if everyone would like to pull all hash algorithm.

Thanks,
Qu
>  	select ZLIB_INFLATE
>  	select ZLIB_DEFLATE
>  	select LZO_COMPRESS
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 099401f5dd47..b34f22e55304 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -90,6 +90,7 @@ static const struct btrfs_csums {
>  	const char	*name;
>  } btrfs_csums[] = {
>  	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"),
> +	BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_XXHASH, 8, "xxhash64"),
>  };
>  
>  #define BTRFS_EMPTY_DIR_SIZE 0
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index 5f7ee70b3d1a..54a8ef489850 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -351,6 +351,7 @@ static bool btrfs_supported_super_csum(u16 csum_type)
>  {
>  	switch (csum_type) {
>  	case BTRFS_CSUM_TYPE_CRC32:
> +	case BTRFS_CSUM_TYPE_XXHASH:
>  		return true;
>  	default:
>  		return false;
> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
> index 78de9d5d80c6..7312f675d702 100644
> --- a/fs/btrfs/super.c
> +++ b/fs/btrfs/super.c
> @@ -2464,3 +2464,4 @@ module_exit(exit_btrfs_fs)
>  
>  MODULE_LICENSE("GPL");
>  MODULE_SOFTDEP("pre: crc32c");
> +MODULE_SOFTDEP("pre: xxhash64");
> diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
> index babef98181a2..af4f5dec10b7 100644
> --- a/include/uapi/linux/btrfs_tree.h
> +++ b/include/uapi/linux/btrfs_tree.h
> @@ -302,6 +302,7 @@
>  /* csum types */
>  enum btrfs_csum_type {
>  	BTRFS_CSUM_TYPE_CRC32	= 0,
> +	BTRFS_CSUM_TYPE_XXHASH	= 1,
>  };
>  
>  /*
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH 3/4] btrfs: use xxhash64 for checksumming

[Johannes Thumshirn]

On Thu, Jul 25, 2019 at 08:02:12PM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/7/25 下午5:33, Johannes Thumshirn wrote:
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  fs/btrfs/Kconfig                | 1 +
> >  fs/btrfs/ctree.h                | 1 +
> >  fs/btrfs/disk-io.c              | 1 +
> >  fs/btrfs/super.c                | 1 +
> >  include/uapi/linux/btrfs_tree.h | 1 +
> >  5 files changed, 5 insertions(+)
> > 
> > diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
> > index 38651fae7f21..6d5a01c57da3 100644
> > --- a/fs/btrfs/Kconfig
> > +++ b/fs/btrfs/Kconfig
> > @@ -5,6 +5,7 @@ config BTRFS_FS
> >  	select CRYPTO
> >  	select CRYPTO_CRC32C
> >  	select LIBCRC32C
> > +	select CRYPTO_XXHASH
> 
> Just an off topic idea, can we make such CRYPTO_* support configurable?
> E.g. make something like CONFIG_BTRFS_CRYPTO_XXHASH.
> 
> Not sure if everyone would like to pull all hash algorithm.
 
This is something I thought about as well, but I was afraid of people shooting
themselves in the foot if they forget to switch them on and mkfs with the
wrong option.

Not sure what's the better way here? Dave?

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [RFC PATCH 3/4] btrfs: use xxhash64 for checksumming

[David Sterba]

On Thu, Jul 25, 2019 at 04:18:37PM +0200, Johannes Thumshirn wrote:
> On Thu, Jul 25, 2019 at 08:02:12PM +0800, Qu Wenruo wrote:
> > 
> > 
> > On 2019/7/25 下午5:33, Johannes Thumshirn wrote:
> > > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > > ---
> > >  fs/btrfs/Kconfig                | 1 +
> > >  fs/btrfs/ctree.h                | 1 +
> > >  fs/btrfs/disk-io.c              | 1 +
> > >  fs/btrfs/super.c                | 1 +
> > >  include/uapi/linux/btrfs_tree.h | 1 +
> > >  5 files changed, 5 insertions(+)
> > > 
> > > diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
> > > index 38651fae7f21..6d5a01c57da3 100644
> > > --- a/fs/btrfs/Kconfig
> > > +++ b/fs/btrfs/Kconfig
> > > @@ -5,6 +5,7 @@ config BTRFS_FS
> > >  	select CRYPTO
> > >  	select CRYPTO_CRC32C
> > >  	select LIBCRC32C
> > > +	select CRYPTO_XXHASH
> > 
> > Just an off topic idea, can we make such CRYPTO_* support configurable?
> > E.g. make something like CONFIG_BTRFS_CRYPTO_XXHASH.
> > 
> > Not sure if everyone would like to pull all hash algorithm.
>  
> This is something I thought about as well, but I was afraid of people shooting
> themselves in the foot if they forget to switch them on and mkfs with the
> wrong option.

> Not sure what's the better way here? Dave?

I'd go to pull everything unconditionally, which means that all
dependent modules are built and does not require the user to tweak the
config.

I understand that there are setups that don't want to provide all hash
algorithms eg. due to space constraints. That will be still possible and
puts the "burden" on the distributor/integrator. Simply don't provide
the .ko files. The crypto API can detect that during mount the shash
descriptor cannot be instantiated and will fail. In specialized setups
this is ok, because lack of the hash algorithm is known.

For everybody else, the filesystem should come with all parts included
so the features work.

The situation with zlib/lzo/zstd is different because we use the library
functions, not the separate .ko modules. This is a bit more convenient.

We don't want to do that with the hash algorithms though because there
are usually optimized verions that we do want to use, and the crypto API
does all the work.

[RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

Export supported checksum algorithms via sysfs.

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 fs/btrfs/sysfs.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 9539f8143b7a..920282a3452b 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -182,6 +182,30 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
 	return mode;
 }
 
+static ssize_t btrfs_checksums_show(struct kobject *kobj,
+				       struct kobj_attribute *a, char *buf)
+{
+	ssize_t ret = 0;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(btrfs_csums); i++) {
+		ret += snprintf(buf + ret, PAGE_SIZE, "%s%s",
+				(i == 0 ? "" : ", "),
+				btrfs_csums[i].name);
+
+	}
+
+	ret += snprintf(buf + ret, PAGE_SIZE, "\n");
+	return ret;
+}
+
+static ssize_t btrfs_checksums_store(struct kobject *kobj,
+					struct kobj_attribute *a,
+					const char *buf, size_t count)
+{
+	return -EPERM;
+}
+
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
 BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
 BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
@@ -195,6 +219,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
 BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
 BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
 
+static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
+	.kobj_attr = __INIT_KOBJ_ATTR(checksums, S_IRUGO,
+				      btrfs_checksums_show,
+				      btrfs_checksums_store),
+	.feature_set	= FEAT_INCOMPAT,
+	.feature_bit	= 0,
+};
+
 static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(mixed_backref),
 	BTRFS_FEAT_ATTR_PTR(default_subvol),
@@ -208,6 +240,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(no_holes),
 	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
 	BTRFS_FEAT_ATTR_PTR(free_space_tree),
+
+	&btrfs_attr_features_checksums_name.kobj_attr.attr,
+
 	NULL
 };
 
-- 
2.16.4

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[David Sterba]

On Thu, Jul 25, 2019 at 11:33:51AM +0200, Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> Export supported checksum algorithms via sysfs.

I wonder if we should also export the implementation that would be used.
This could be crross referenced with /proc/crypto, but having it in one
place would be IMHO convenient.  Also for the case when the kernel
module is missing.

Currently the hash names are printed as comma separated values so we'd
need bit something structured:

crc32c: crc32c-intel
xxhash64: xxhash-generic

or if there's some other format in common use.

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[Johannes Thumshirn]

On Tue, Jul 30, 2019 at 07:19:04PM +0200, David Sterba wrote:
> On Thu, Jul 25, 2019 at 11:33:51AM +0200, Johannes Thumshirn wrote:
> > From: David Sterba <dsterba@suse.com>
> > 
> > Export supported checksum algorithms via sysfs.
> 
> I wonder if we should also export the implementation that would be used.
> This could be crross referenced with /proc/crypto, but having it in one
> place would be IMHO convenient.  Also for the case when the kernel
> module is missing.
> 
> Currently the hash names are printed as comma separated values so we'd
> need bit something structured:
> 
> crc32c: crc32c-intel
> xxhash64: xxhash-generic
> 
> or if there's some other format in common use.

Sounds good, will do.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[Johannes Thumshirn]

On Tue, Jul 30, 2019 at 07:19:04PM +0200, David Sterba wrote:
> On Thu, Jul 25, 2019 at 11:33:51AM +0200, Johannes Thumshirn wrote:
> > From: David Sterba <dsterba@suse.com>
> > 
> > Export supported checksum algorithms via sysfs.
> 
> I wonder if we should also export the implementation that would be used.
> This could be crross referenced with /proc/crypto, but having it in one
> place would be IMHO convenient.  Also for the case when the kernel
> module is missing.
> 
> Currently the hash names are printed as comma separated values so we'd
> need bit something structured:
> 
> crc32c: crc32c-intel
> xxhash64: xxhash-generic

I thought a bit more about it and it's not quite that easy as I would need to
have access to the respective "struct crypto_shash". For the currently used
checksum this isn't much of a problem, as I get it via "struct btrfs_fs_info".

But this sysfs attribute lists all the checksumming algorithms the current
btrfs version is able to use irrespectively of the currently mounted
file-systems.

So yes I can do it but I think this is for another sysfs attribute which shows
the used checksumming algorithm for this FS, not the supported checksumming
algorithms.

Byte,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[David Sterba]

On Wed, Aug 07, 2019 at 04:10:05PM +0200, Johannes Thumshirn wrote:
> On Tue, Jul 30, 2019 at 07:19:04PM +0200, David Sterba wrote:
> > On Thu, Jul 25, 2019 at 11:33:51AM +0200, Johannes Thumshirn wrote:
> > > From: David Sterba <dsterba@suse.com>
> > > 
> > > Export supported checksum algorithms via sysfs.
> > 
> > I wonder if we should also export the implementation that would be used.
> > This could be crross referenced with /proc/crypto, but having it in one
> > place would be IMHO convenient.  Also for the case when the kernel
> > module is missing.
> > 
> > Currently the hash names are printed as comma separated values so we'd
> > need bit something structured:
> > 
> > crc32c: crc32c-intel
> > xxhash64: xxhash-generic
> 
> I thought a bit more about it and it's not quite that easy as I would need to
> have access to the respective "struct crypto_shash". For the currently used
> checksum this isn't much of a problem, as I get it via "struct btrfs_fs_info".
> 
> But this sysfs attribute lists all the checksumming algorithms the current
> btrfs version is able to use irrespectively of the currently mounted
> file-systems.
> 
> So yes I can do it but I think this is for another sysfs attribute which shows
> the used checksumming algorithm for this FS, not the supported checksumming
> algorithms.

I see. We can't know which implementation would be used until we
actually try to use it, so two sysfs files would make more sense. Plain
list for all supported algos and one with the implementation for a given
mounted filesystem.

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> Export supported checksum algorithms via sysfs.
> 
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  fs/btrfs/sysfs.c | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
> 
> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> index 9539f8143b7a..920282a3452b 100644
> --- a/fs/btrfs/sysfs.c
> +++ b/fs/btrfs/sysfs.c
> @@ -182,6 +182,30 @@ static umode_t btrfs_feature_visible(struct kobject *kobj,
>  	return mode;
>  }
>  
> +static ssize_t btrfs_checksums_show(struct kobject *kobj,
> +				       struct kobj_attribute *a, char *buf)
> +{
> +	ssize_t ret = 0;
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(btrfs_csums); i++) {
> +		ret += snprintf(buf + ret, PAGE_SIZE, "%s%s",
> +				(i == 0 ? "" : ", "),
> +				btrfs_csums[i].name);
> +
> +	}
> +
> +	ret += snprintf(buf + ret, PAGE_SIZE, "\n");
> +	return ret;
> +}
> +
> +static ssize_t btrfs_checksums_store(struct kobject *kobj,
> +					struct kobj_attribute *a,
> +					const char *buf, size_t count)
> +{
> +	return -EPERM;
> +}
> +
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
>  BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
>  BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
> @@ -195,6 +219,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
>  BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
>  BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
>  
> +static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
> +	.kobj_attr = __INIT_KOBJ_ATTR(checksums, S_IRUGO,
> +				      btrfs_checksums_show,
> +				      btrfs_checksums_store),

Since we won't ever support writing to this sysfs just kill
btrfs_checksums_store and simply pass NULL as the last argument to
INIT_KOBJ_ATTR.

> +	.feature_set	= FEAT_INCOMPAT,
> +	.feature_bit	= 0,
> +};
> +
>  static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(mixed_backref),
>  	BTRFS_FEAT_ATTR_PTR(default_subvol),
> @@ -208,6 +240,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
>  	BTRFS_FEAT_ATTR_PTR(no_holes),
>  	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
>  	BTRFS_FEAT_ATTR_PTR(free_space_tree),
> +
> +	&btrfs_attr_features_checksums_name.kobj_attr.attr,
> +
>  	NULL
>  };
>  
> 

Re: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums

[Johannes Thumshirn]

On Mon, Aug 12, 2019 at 12:19:13PM +0300, Nikolay Borisov wrote:
> > +static struct btrfs_feature_attr btrfs_attr_features_checksums_name = {
> > +	.kobj_attr = __INIT_KOBJ_ATTR(checksums, S_IRUGO,
> > +				      btrfs_checksums_show,
> > +				      btrfs_checksums_store),
> 
> Since we won't ever support writing to this sysfs just kill
> btrfs_checksums_store and simply pass NULL as the last argument to
> INIT_KOBJ_ATTR.
> 

Yup will do.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[RFC PATCH 05/17] btrfs-progs: add option for checksum type to mkfs

[Johannes Thumshirn]

Add an option to mkfs to specify which checksum algorithm will be used for
the filesystem.

XXX this patch should go last in the series.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 mkfs/common.c |  2 +-
 mkfs/common.h |  2 ++
 mkfs/main.c   | 21 ++++++++++++++++++++-
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/mkfs/common.c b/mkfs/common.c
index caca5e707233..8249492704ad 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -201,7 +201,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
 	btrfs_set_super_nodesize(&super, cfg->nodesize);
 	btrfs_set_super_stripesize(&super, cfg->stripesize);
-	btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32);
+	btrfs_set_super_csum_type(&super, cfg->csum_type);
 	btrfs_set_super_chunk_root_generation(&super, 1);
 	btrfs_set_super_cache_generation(&super, -1);
 	btrfs_set_super_incompat_flags(&super, cfg->features);
diff --git a/mkfs/common.h b/mkfs/common.h
index 28912906d0a9..384c1ef6a753 100644
--- a/mkfs/common.h
+++ b/mkfs/common.h
@@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
 	u64 features;
 	/* Size of the filesystem in bytes */
 	u64 num_bytes;
+	/* checksum algorithm to use */
+	u16 csum_type;
 
 	/* Output fields, set during creation */
 
diff --git a/mkfs/main.c b/mkfs/main.c
index 8dbec0717b89..3deff76045ba 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -346,6 +346,7 @@ static void print_usage(int ret)
 	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
 	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
 	printf("\t-f|--force              force overwrite of existing filesystem\n");
+	printf("\t-C|--checksum           checksum algorithm to use (default: crc32c)\n");
 	printf("  general:\n");
 	printf("\t-q|--quiet              no messages except errors\n");
 	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
@@ -380,6 +381,18 @@ static u64 parse_profile(const char *s)
 	return 0;
 }
 
+static u16 parse_csum_type(const char *s)
+{
+	if (strcasecmp(s, "crc32c") == 0) {
+		return BTRFS_CSUM_TYPE_CRC32;
+	} else {
+		error("unknown csum type %s", s);
+		exit(1);
+	}
+	/* not reached */
+	return 0;
+}
+
 static char *parse_label(const char *input)
 {
 	int len = strlen(input);
@@ -826,6 +839,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
 	struct mkfs_allocation allocation = { 0 };
 	struct btrfs_mkfs_config mkfs_cfg;
+	int csum_type = BTRFS_CSUM_TYPE_CRC32;
 
 	crc32c_optimization_init();
 
@@ -835,6 +849,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 		static const struct option long_options[] = {
 			{ "alloc-start", required_argument, NULL, 'A'},
 			{ "byte-count", required_argument, NULL, 'b' },
+			{ "checksum", required_argument, NULL, 'C' },
 			{ "force", no_argument, NULL, 'f' },
 			{ "leafsize", required_argument, NULL, 'l' },
 			{ "label", required_argument, NULL, 'L'},
@@ -854,7 +869,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			{ NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+		c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq",
 				long_options, NULL);
 		if (c < 0)
 			break;
@@ -932,6 +947,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			case GETOPT_VAL_SHRINK:
 				shrink_rootdir = true;
 				break;
+			case 'C':
+				csum_type = parse_csum_type(optarg);
+				break;
 			case GETOPT_VAL_HELP:
 			default:
 				print_usage(c != GETOPT_VAL_HELP);
@@ -1170,6 +1188,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	mkfs_cfg.sectorsize = sectorsize;
 	mkfs_cfg.stripesize = stripesize;
 	mkfs_cfg.features = features;
+	mkfs_cfg.csum_type = csum_type;
 
 	ret = make_btrfs(fd, &mkfs_cfg);
 	if (ret) {
-- 
2.16.4

Re: [RFC PATCH 05/17] btrfs-progs: add option for checksum type to mkfs

[David Sterba]

On Thu, Jul 25, 2019 at 11:33:52AM +0200, Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
> 
> XXX this patch should go last in the series.

If the code update and the mkfs option are split, I can merge the
preparatory part independently. In case you're going to respin the
series, please do 2 patches and sort the preparatory patches to the
beginning of the series.

Re: [RFC PATCH 05/17] btrfs-progs: add option for checksum type to mkfs

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
> 
> XXX this patch should go last in the series.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

<snip>

> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -346,6 +346,7 @@ static void print_usage(int ret)
>  	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
>  	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
>  	printf("\t-f|--force              force overwrite of existing filesystem\n");
> +	printf("\t-C|--checksum           checksum algorithm to use (default: crc32c)\n");
>  	printf("  general:\n");
>  	printf("\t-q|--quiet              no messages except errors\n");
>  	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
> @@ -380,6 +381,18 @@ static u64 parse_profile(const char *s)
>  	return 0;
>  }
>  
> +static u16 parse_csum_type(const char *s)
> +{
> +	if (strcasecmp(s, "crc32c") == 0) {
> +		return BTRFS_CSUM_TYPE_CRC32;
> +	} else {
> +		error("unknown csum type %s", s);
> +		exit(1);
> +	}
> +	/* not reached */
> +	return 0;
> +}

How hard would it be making this function return 'enum btrfs_csum_type'
and all further references to the checksum type be done through this
enum type? Functionally this won't bring any differences but will make
the code very explicit. Perhaps you'd have to also add a value for
invalid checksum ?

<snip>

[RFC PATCH 06/17] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[Johannes Thumshirn]

The callers of csum_tree_block_size() blindly assume we're only having
crc32c as a possible checksum and thus pass in
btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
csum_tree_block_size().

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 mkfs/common.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mkfs/common.c b/mkfs/common.c
index 8249492704ad..d63a9267bca3 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 
 	/* write back root tree */
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
 	if (ret != BTRFS_SUPER_INFO_SIZE) {
-- 
2.16.4

Re: [RFC PATCH 06/17] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

[RFC PATCH 07/17] btrfs-progs: use btrfs_csum_data() in __csum_tree_block_size()

[Johannes Thumshirn]

Use the btrfs_csum_data() wrapper in __csum_tree_block_size() instead of
directly calling crc32c().

This helps us when plumbing new checksum algorithms into the FS.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 disk-io.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/disk-io.c b/disk-io.c
index be44eead5cef..01314504a50a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = crc32c(crc, buf->data + BTRFS_CSUM_SIZE, len);
+	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len);
 	btrfs_csum_final(crc, result);
 
 	if (verify) {
-- 
2.16.4

Re: [RFC PATCH 07/17] btrfs-progs: use btrfs_csum_data() in __csum_tree_block_size()

[David Sterba]

On Thu, Jul 25, 2019 at 11:33:54AM +0200, Johannes Thumshirn wrote:
> Use the btrfs_csum_data() wrapper in __csum_tree_block_size() instead of
> directly calling crc32c().
> 
> This helps us when plumbing new checksum algorithms into the FS.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Added to devel.

[RFC PATCH 08/17] btrfs-progs: cache csum_type in recover_control

[Johannes Thumshirn]

Cache the super-block's checksum type field in 'struct recover_control'.
This will be needed for further refactoring the checksum handling.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 cmds/rescue-chunk-recover.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 608af9d49407..308731ea5ea6 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -47,6 +47,7 @@ struct recover_control {
 	int yes;
 
 	u16 csum_size;
+	u16 csum_type;
 	u32 sectorsize;
 	u32 nodesize;
 	u64 generation;
@@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path)
 	rc->generation = btrfs_super_generation(sb);
 	rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb);
 	rc->csum_size = btrfs_super_csum_size(sb);
+	rc->csum_type = btrfs_super_csum_type(sb);
 
 	/* if seed, the result of scanning below will be partial */
 	if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) {
-- 
2.16.4

Re: [RFC PATCH 08/17] btrfs-progs: cache csum_type in recover_control

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> Cache the super-block's checksum type field in 'struct recover_control'.
> This will be needed for further refactoring the checksum handling.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---

Reviewed-by: Nikolay Borisov <nborisov@suse.com>


>  cmds/rescue-chunk-recover.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 608af9d49407..308731ea5ea6 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -47,6 +47,7 @@ struct recover_control {
>  	int yes;
>  
>  	u16 csum_size;
> +	u16 csum_type;
>  	u32 sectorsize;
>  	u32 nodesize;
>  	u64 generation;
> @@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path)
>  	rc->generation = btrfs_super_generation(sb);
>  	rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb);
>  	rc->csum_size = btrfs_super_csum_size(sb);
> +	rc->csum_type = btrfs_super_csum_type(sb);
>  
>  	/* if seed, the result of scanning below will be partial */
>  	if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) {
> 

[RFC PATCH 09/17] progs: pass in a btrfs_mkfs_config to write_temp_extent_buffer

[Johannes Thumshirn]

Pass in a btrfs_mkfs_config to write_temp_extent_buffer(), this is needed
so we can grab the checksum type for checksum buffer verification in later
patches.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 convert/common.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/convert/common.c b/convert/common.c
index af4f8d372299..dea5f5b20d50 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -218,7 +218,8 @@ static void insert_temp_root_item(struct extent_buffer *buf,
  * Setup an extent buffer for tree block.
  */
 static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
-					   u64 bytenr)
+					   u64 bytenr,
+					   struct btrfs_mkfs_config *cfg)
 {
 	int ret;
 
@@ -281,7 +282,7 @@ static int setup_temp_root_tree(int fd, struct btrfs_mkfs_config *cfg,
 	insert_temp_root_item(buf, cfg, &slot, &itemoff,
 			      BTRFS_CSUM_TREE_OBJECTID, csum_bytenr);
 
-	ret = write_temp_extent_buffer(fd, buf, root_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, root_bytenr, cfg);
 out:
 	free(buf);
 	return ret;
@@ -456,7 +457,7 @@ static int setup_temp_chunk_tree(int fd, struct btrfs_mkfs_config *cfg,
 				     BTRFS_BLOCK_GROUP_METADATA);
 	if (ret < 0)
 		goto out;
-	ret = write_temp_extent_buffer(fd, buf, chunk_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, chunk_bytenr, cfg);
 
 out:
 	free(buf);
@@ -515,7 +516,7 @@ static int setup_temp_dev_tree(int fd, struct btrfs_mkfs_config *cfg,
 			       BTRFS_MKFS_SYSTEM_GROUP_SIZE);
 	insert_temp_dev_extent(buf, &slot, &itemoff, meta_chunk_start,
 			       BTRFS_CONVERT_META_GROUP_SIZE);
-	ret = write_temp_extent_buffer(fd, buf, dev_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, dev_bytenr, cfg);
 out:
 	free(buf);
 	return ret;
@@ -537,7 +538,7 @@ static int setup_temp_fs_tree(int fd, struct btrfs_mkfs_config *cfg,
 	/*
 	 * Temporary fs tree is completely empty.
 	 */
-	ret = write_temp_extent_buffer(fd, buf, fs_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, fs_bytenr, cfg);
 out:
 	free(buf);
 	return ret;
@@ -559,7 +560,7 @@ static int setup_temp_csum_tree(int fd, struct btrfs_mkfs_config *cfg,
 	/*
 	 * Temporary csum tree is completely empty.
 	 */
-	ret = write_temp_extent_buffer(fd, buf, csum_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, csum_bytenr, cfg);
 out:
 	free(buf);
 	return ret;
@@ -765,7 +766,7 @@ static int setup_temp_extent_tree(int fd, struct btrfs_mkfs_config *cfg,
 	if (ret < 0)
 		goto out;
 
-	ret = write_temp_extent_buffer(fd, buf, extent_bytenr);
+	ret = write_temp_extent_buffer(fd, buf, extent_bytenr, cfg);
 out:
 	free(buf);
 	return ret;
-- 
2.16.4

Re: [RFC PATCH 09/17] progs: pass in a btrfs_mkfs_config to write_temp_extent_buffer

[David Sterba]

On Thu, Jul 25, 2019 at 11:33:56AM +0200, Johannes Thumshirn wrote:
> Pass in a btrfs_mkfs_config to write_temp_extent_buffer(), this is needed
> so we can grab the checksum type for checksum buffer verification in later
> patches.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Added to devel.

Re: [RFC PATCH 09/17] progs: pass in a btrfs_mkfs_config to write_temp_extent_buffer

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> Pass in a btrfs_mkfs_config to write_temp_extent_buffer(), this is needed
> so we can grab the checksum type for checksum buffer verification in later
> patches.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  convert/common.c | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/convert/common.c b/convert/common.c
> index af4f8d372299..dea5f5b20d50 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -218,7 +218,8 @@ static void insert_temp_root_item(struct extent_buffer *buf,
>   * Setup an extent buffer for tree block.
>   */
>  static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
> -					   u64 bytenr)
> +					   u64 bytenr,
> +					   struct btrfs_mkfs_config *cfg)
>  {
>  	int ret;
>  
> @@ -281,7 +282,7 @@ static int setup_temp_root_tree(int fd, struct btrfs_mkfs_config *cfg,
>  	insert_temp_root_item(buf, cfg, &slot, &itemoff,
>  			      BTRFS_CSUM_TREE_OBJECTID, csum_bytenr);
>  
> -	ret = write_temp_extent_buffer(fd, buf, root_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, root_bytenr, cfg);
>  out:
>  	free(buf);
>  	return ret;
> @@ -456,7 +457,7 @@ static int setup_temp_chunk_tree(int fd, struct btrfs_mkfs_config *cfg,
>  				     BTRFS_BLOCK_GROUP_METADATA);
>  	if (ret < 0)
>  		goto out;
> -	ret = write_temp_extent_buffer(fd, buf, chunk_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, chunk_bytenr, cfg);
>  
>  out:
>  	free(buf);
> @@ -515,7 +516,7 @@ static int setup_temp_dev_tree(int fd, struct btrfs_mkfs_config *cfg,
>  			       BTRFS_MKFS_SYSTEM_GROUP_SIZE);
>  	insert_temp_dev_extent(buf, &slot, &itemoff, meta_chunk_start,
>  			       BTRFS_CONVERT_META_GROUP_SIZE);
> -	ret = write_temp_extent_buffer(fd, buf, dev_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, dev_bytenr, cfg);
>  out:
>  	free(buf);
>  	return ret;
> @@ -537,7 +538,7 @@ static int setup_temp_fs_tree(int fd, struct btrfs_mkfs_config *cfg,
>  	/*
>  	 * Temporary fs tree is completely empty.
>  	 */
> -	ret = write_temp_extent_buffer(fd, buf, fs_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, fs_bytenr, cfg);
>  out:
>  	free(buf);
>  	return ret;
> @@ -559,7 +560,7 @@ static int setup_temp_csum_tree(int fd, struct btrfs_mkfs_config *cfg,
>  	/*
>  	 * Temporary csum tree is completely empty.
>  	 */
> -	ret = write_temp_extent_buffer(fd, buf, csum_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, csum_bytenr, cfg);
>  out:
>  	free(buf);
>  	return ret;
> @@ -765,7 +766,7 @@ static int setup_temp_extent_tree(int fd, struct btrfs_mkfs_config *cfg,
>  	if (ret < 0)
>  		goto out;
>  
> -	ret = write_temp_extent_buffer(fd, buf, extent_bytenr);
> +	ret = write_temp_extent_buffer(fd, buf, extent_bytenr, cfg);
>  out:
>  	free(buf);
>  	return ret;
> 

[RFC PATCH 10/17] btrfs-progs: add checksum type to checksumming functions

[Johannes Thumshirn]

Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
and verify_tree_block_csum_silent().

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-corrupt-block.c       |  3 ++-
 cmds/rescue-chunk-recover.c |  3 ++-
 convert/common.c            |  3 ++-
 convert/main.c              |  3 ++-
 disk-io.c                   | 21 ++++++++++++---------
 disk-io.h                   |  5 +++--
 mkfs/common.c               | 21 ++++++++++++++-------
 7 files changed, 37 insertions(+), 22 deletions(-)

diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index bbef0c02e5d1..1dde9594bdcc 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
 	if (!trans) {
 		u16 csum_size =
 			btrfs_super_csum_size(fs_info->super_copy);
-		csum_tree_block_size(eb, csum_size, 0);
+		u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+		csum_tree_block_size(eb, csum_size, 0, csum_type);
 		write_extent_to_disk(eb);
 	}
 }
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 308731ea5ea6..1a368310d895 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
 			continue;
 		}
 
-		if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
+		if (verify_tree_block_csum_silent(buf, rc->csum_size,
+						  rc->csum_type)) {
 			bytenr += rc->sectorsize;
 			continue;
 		}
diff --git a/convert/common.c b/convert/common.c
index dea5f5b20d50..f8bbb23cba89 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
 {
 	int ret;
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 
 	/* Temporary extent buffer is always mapped 1:1 on disk */
 	ret = pwrite(fd, buf->data, buf->len, bytenr);
diff --git a/convert/main.c b/convert/main.c
index 9711874bd137..5e6b12431f59 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
 	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
 	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+			     btrfs_super_csum_type(super));
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 		BTRFS_SUPER_INFO_OFFSET);
 	if (ret != BTRFS_SUPER_INFO_SIZE)
diff --git a/disk-io.c b/disk-io.c
index 01314504a50a..a4995a628210 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
-				  int verify, int silent)
+				  int verify, int silent, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 len;
@@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	return 0;
 }
 
-int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
+int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
+			 u16 csum_type)
 {
-	return __csum_tree_block_size(buf, csum_size, verify, 0);
+	return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
 }
 
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+				  u16 csum_type)
 {
-	return __csum_tree_block_size(buf, csum_size, 1, 1);
+	return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
 }
 
 int csum_tree_block(struct btrfs_fs_info *fs_info,
 		    struct extent_buffer *buf, int verify)
 {
-	u16 csum_size =
-		btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
 	if (verify && fs_info->suppress_check_block_errors)
-		return verify_tree_block_csum_silent(buf, csum_size);
-	return csum_tree_block_size(buf, csum_size, verify);
+		return verify_tree_block_csum_silent(buf, csum_size, csum_type);
+	return csum_tree_block_size(buf, csum_size, verify, csum_type);
 }
 
 struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
diff --git a/disk-io.h b/disk-io.h
index 7b5c3806ba98..394997ad72cb 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
-			 int verify);
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
+			 int verify, u16 csum_type);
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+				  u16 csum_type);
 int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
 int write_tree_block(struct btrfs_trans_handle *trans,
 		     struct btrfs_fs_info *fs_info,
diff --git a/mkfs/common.c b/mkfs/common.c
index d63a9267bca3..4a417bd7a306 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 
 	/* write back root tree */
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
 	if (ret != BTRFS_SUPER_INFO_SIZE) {
-- 
2.16.4

Re: [RFC PATCH 10/17] btrfs-progs: add checksum type to checksumming functions

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
> and verify_tree_block_csum_silent().
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  btrfs-corrupt-block.c       |  3 ++-
>  cmds/rescue-chunk-recover.c |  3 ++-
>  convert/common.c            |  3 ++-
>  convert/main.c              |  3 ++-
>  disk-io.c                   | 21 ++++++++++++---------
>  disk-io.h                   |  5 +++--
>  mkfs/common.c               | 21 ++++++++++++++-------
>  7 files changed, 37 insertions(+), 22 deletions(-)
> 
> diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
> index bbef0c02e5d1..1dde9594bdcc 100644
> --- a/btrfs-corrupt-block.c
> +++ b/btrfs-corrupt-block.c
> @@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
>  	if (!trans) {
>  		u16 csum_size =
>  			btrfs_super_csum_size(fs_info->super_copy);
> -		csum_tree_block_size(eb, csum_size, 0);
> +		u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
> +		csum_tree_block_size(eb, csum_size, 0, csum_type);

I'm not a big fan of open-coding this csum manipulation. Generally if we
ensure that we always pass a well-formed eb to csum_tree_block_size and
by well-formed I mean one which has its ->fs_info, which in turn is
well-formed then we can encapsulate this interrogation of csum type into
csum_tree_block.

>  		write_extent_to_disk(eb);
>  	}
>  }
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 308731ea5ea6..1a368310d895 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
>  			continue;
>  		}
>  
> -		if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
> +		if (verify_tree_block_csum_silent(buf, rc->csum_size,
> +						  rc->csum_type)) {

But this is not a well-formed eb :(

>  			bytenr += rc->sectorsize;
>  			continue;
>  		}
> diff --git a/convert/common.c b/convert/common.c
> index dea5f5b20d50..f8bbb23cba89 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
>  {
>  	int ret;
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);

Neither is this ...
>  
>  	/* Temporary extent buffer is always mapped 1:1 on disk */
>  	ret = pwrite(fd, buf->data, buf->len, bytenr);
> diff --git a/convert/main.c b/convert/main.c
> index 9711874bd137..5e6b12431f59 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
>  	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
>  	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> +			     btrfs_super_csum_type(super));
>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>  		BTRFS_SUPER_INFO_OFFSET);
>  	if (ret != BTRFS_SUPER_INFO_SIZE)
> diff --git a/disk-io.c b/disk-io.c
> index 01314504a50a..a4995a628210 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
>  }
>  
>  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> -				  int verify, int silent)
> +				  int verify, int silent, u16 csum_type)
>  {
>  	u8 result[BTRFS_CSUM_SIZE];
>  	u32 len;
> @@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>  	return 0;
>  }
>  
> -int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
> +int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
> +			 u16 csum_type)
>  {
> -	return __csum_tree_block_size(buf, csum_size, verify, 0);
> +	return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
>  }
>  
> -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
> +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
> +				  u16 csum_type)
>  {
> -	return __csum_tree_block_size(buf, csum_size, 1, 1);
> +	return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
>  }

Now that those function take the csum_type it renders passing the
csum_size redundant. Refactor them so that only csum_type is passed and
make __csum_tree_block_size (perhahps it should be renamed now?) get the
size based on the type.

I think the csum interface can be simplified further:

1. Leave csum_tree_block for callers that pass well-formed eb's (as is
currently)

2. Rename __csum_tree_block_size to __csum_tree_block which takes just
the buffer, boolean flag verify and u16 (or better enum btrfs_csum_type)
csum_type arguments and the nitty-gritty details

3. Remove csum_tree_block_size and directly call the new
__csum_tree_block. The former just wraps the 'silent' parameter...

In the end we will be left with just 2 functions:

(__)csum_tree_blocks .

Additionally verify_tree_block_csum_silent has really one caller because
in csum_tree_block it's enough to call __csum_tree_block_size just once
and pass verify and fs_info->suppress_check_block_errors directly to it.
Then you can also directly call __csum_tree_block in scan_one_device in
rescue-chunk-recover.c . And this would put some sanity into the current
csum interfaces in progs.

I guess this could land as a follow-up to your patches...

>  
>  int csum_tree_block(struct btrfs_fs_info *fs_info,
>  		    struct extent_buffer *buf, int verify)
>  {
> -	u16 csum_size =
> -		btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
> +
>  	if (verify && fs_info->suppress_check_block_errors)
> -		return verify_tree_block_csum_silent(buf, csum_size);
> -	return csum_tree_block_size(buf, csum_size, verify);
> +		return verify_tree_block_csum_silent(buf, csum_size, csum_type);
> +	return csum_tree_block_size(buf, csum_size, verify, csum_type);
>  }
>  
>  struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
> diff --git a/disk-io.h b/disk-io.h
> index 7b5c3806ba98..394997ad72cb 100644
> --- a/disk-io.h
> +++ b/disk-io.h
> @@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
>  
>  int btrfs_open_device(struct btrfs_device *dev);
>  int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
> -			 int verify);
> -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
> +			 int verify, u16 csum_type);
> +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
> +				  u16 csum_type);
>  int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
>  int write_tree_block(struct btrfs_trans_handle *trans,
>  		     struct btrfs_fs_info *fs_info,
> diff --git a/mkfs/common.c b/mkfs/common.c
> index d63a9267bca3..4a417bd7a306 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
>  	}
>  
>  	/* generate checksum */
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  
>  	/* write back root tree */
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
> @@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, 0);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, 0);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
>  	memcpy(buf->data, &super, sizeof(super));
>  	buf->len = BTRFS_SUPER_INFO_SIZE;
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>  			cfg->blocks[MKFS_SUPER_BLOCK]);
>  	if (ret != BTRFS_SUPER_INFO_SIZE) {
> 

Re: [RFC PATCH 10/17] btrfs-progs: add checksum type to checksumming functions

[Johannes Thumshirn]

On 12/08/2019 12:21, Nikolay Borisov wrote:
> 
> 
> On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
>> Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
>> and verify_tree_block_csum_silent().
>>
>> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>> ---
>>  btrfs-corrupt-block.c       |  3 ++-
>>  cmds/rescue-chunk-recover.c |  3 ++-
>>  convert/common.c            |  3 ++-
>>  convert/main.c              |  3 ++-
>>  disk-io.c                   | 21 ++++++++++++---------
>>  disk-io.h                   |  5 +++--
>>  mkfs/common.c               | 21 ++++++++++++++-------
>>  7 files changed, 37 insertions(+), 22 deletions(-)
>>
>> diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
>> index bbef0c02e5d1..1dde9594bdcc 100644
>> --- a/btrfs-corrupt-block.c
>> +++ b/btrfs-corrupt-block.c
>> @@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
>>  	if (!trans) {
>>  		u16 csum_size =
>>  			btrfs_super_csum_size(fs_info->super_copy);
>> -		csum_tree_block_size(eb, csum_size, 0);
>> +		u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
>> +		csum_tree_block_size(eb, csum_size, 0, csum_type);
> 
> I'm not a big fan of open-coding this csum manipulation. Generally if we
> ensure that we always pass a well-formed eb to csum_tree_block_size and
> by well-formed I mean one which has its ->fs_info, which in turn is
> well-formed then we can encapsulate this interrogation of csum type into
> csum_tree_block.
> 
>>  		write_extent_to_disk(eb);
>>  	}
>>  }
>> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
>> index 308731ea5ea6..1a368310d895 100644
>> --- a/cmds/rescue-chunk-recover.c
>> +++ b/cmds/rescue-chunk-recover.c
>> @@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
>>  			continue;
>>  		}
>>  
>> -		if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
>> +		if (verify_tree_block_csum_silent(buf, rc->csum_size,
>> +						  rc->csum_type)) {
> 
> But this is not a well-formed eb :(
> 
>>  			bytenr += rc->sectorsize;
>>  			continue;
>>  		}
>> diff --git a/convert/common.c b/convert/common.c
>> index dea5f5b20d50..f8bbb23cba89 100644
>> --- a/convert/common.c
>> +++ b/convert/common.c
>> @@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
>>  {
>>  	int ret;
>>  
>> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
>> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
>> +			     cfg->csum_type);
> 
> Neither is this ...
>>  
>>  	/* Temporary extent buffer is always mapped 1:1 on disk */
>>  	ret = pwrite(fd, buf->data, buf->len, bytenr);
>> diff --git a/convert/main.c b/convert/main.c
>> index 9711874bd137..5e6b12431f59 100644
>> --- a/convert/main.c
>> +++ b/convert/main.c
>> @@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
>>  	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
>>  	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>>  
>> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
>> +	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
>> +			     btrfs_super_csum_type(super));
>>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>>  		BTRFS_SUPER_INFO_OFFSET);
>>  	if (ret != BTRFS_SUPER_INFO_SIZE)
>> diff --git a/disk-io.c b/disk-io.c
>> index 01314504a50a..a4995a628210 100644
>> --- a/disk-io.c
>> +++ b/disk-io.c
>> @@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
>>  }
>>  
>>  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>> -				  int verify, int silent)
>> +				  int verify, int silent, u16 csum_type)
>>  {
>>  	u8 result[BTRFS_CSUM_SIZE];
>>  	u32 len;
>> @@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>>  	return 0;
>>  }
>>  
>> -int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
>> +int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
>> +			 u16 csum_type)
>>  {
>> -	return __csum_tree_block_size(buf, csum_size, verify, 0);
>> +	return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
>>  }
>>  
>> -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
>> +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
>> +				  u16 csum_type)
>>  {
>> -	return __csum_tree_block_size(buf, csum_size, 1, 1);
>> +	return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
>>  }
> 
> Now that those function take the csum_type it renders passing the
> csum_size redundant. Refactor them so that only csum_type is passed and
> make __csum_tree_block_size (perhahps it should be renamed now?) get the
> size based on the type.
> 
> I think the csum interface can be simplified further:
> 
> 1. Leave csum_tree_block for callers that pass well-formed eb's (as is
> currently)
> 
> 2. Rename __csum_tree_block_size to __csum_tree_block which takes just
> the buffer, boolean flag verify and u16 (or better enum btrfs_csum_type)
> csum_type arguments and the nitty-gritty details
> 
> 3. Remove csum_tree_block_size and directly call the new
> __csum_tree_block. The former just wraps the 'silent' parameter...
> 
> In the end we will be left with just 2 functions:
> 
> (__)csum_tree_blocks .
> 
> Additionally verify_tree_block_csum_silent has really one caller because
> in csum_tree_block it's enough to call __csum_tree_block_size just once
> and pass verify and fs_info->suppress_check_block_errors directly to it.
> Then you can also directly call __csum_tree_block in scan_one_device in
> rescue-chunk-recover.c . And this would put some sanity into the current
> csum interfaces in progs.
> 
> I guess this could land as a follow-up to your patches...
> 

I'd really like to have this rework as a follow up as it introduces a
non-negligible regression potential and can "easily" be done after this
series has landed.


-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[RFC PATCH 11/17] btrfs-progs: don't assume checksums are always 4 bytes

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c              |  9 ++++++---
 check/main.c                |  2 +-
 cmds/inspect-dump-super.c   |  3 ++-
 cmds/rescue-chunk-recover.c |  2 +-
 convert/common.c            |  3 ++-
 disk-io.c                   | 12 ++++++------
 disk-io.h                   |  2 +-
 file-item.c                 |  2 +-
 8 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 16a26f772494..932c2a0432ef 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb)
 	u32 crc = ~(u32)0;
 
 	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
-				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+				(u8 *)&crc,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
 	return !memcmp(sb, &result, csum_size);
@@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb)
 	u32 crc = ~(u32)0;
 
 	if (is_sb) {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+				      (u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	} else {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+				      (u8 *)&crc,
 				BLOCKSIZE - BTRFS_CSUM_SIZE);
 	}
 	btrfs_csum_final(crc, result);
diff --git a/check/main.c b/check/main.c
index bb57933b83fc..92a7e40c1ebf 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5622,7 +5622,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 				tmp = offset + data_checked;
 
 				csum = btrfs_csum_data((char *)data + tmp,
-						csum, fs_info->sectorsize);
+						(u8 *)&csum, fs_info->sectorsize);
 				btrfs_csum_final(csum, (u8 *)&csum);
 
 				csum_offset = leaf_offset +
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 65fb3506eac6..96ad3deca3d8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size)
 	u32 crc = ~(u32)0;
 
 	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
-				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+				(u8 *)&crc,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
 	return !memcmp(sb, &result, csum_size);
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 1a368310d895..0fddb5dd8ead 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(data, csum_result, len);
+	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
 	btrfs_csum_final(csum_result, (u8 *)&csum_result);
 	if (csum_result != tree_csum)
 		ret = 1;
diff --git a/convert/common.c b/convert/common.c
index f8bbb23cba89..ab8e6b9f4749 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
        u32 crc = ~(u32)0;
        int ret;
 
-       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+			     (u8 *)&crc,
                              BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
        btrfs_csum_final(crc, &sb->csum[0]);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
diff --git a/disk-io.c b/disk-io.c
index a4995a628210..a0c37c569d58 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(char *data, u32 seed, size_t len)
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
 {
-	return crc32c(seed, data, len);
+	return crc32c(*(u32*)seed, data, len);
 }
 
 void btrfs_csum_final(u32 crc, u8 *result)
@@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len);
+	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
 	btrfs_csum_final(crc, result);
 
 	if (verify) {
@@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	csum_size = btrfs_csum_sizes[csum_type];
 
 	crc = ~(u32)0;
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
@@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 		btrfs_csum_final(crc, &sb->csum[0]);
 
@@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 		btrfs_set_super_bytenr(sb, bytenr);
 
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 		btrfs_csum_final(crc, &sb->csum[0]);
 
diff --git a/disk-io.h b/disk-io.h
index 394997ad72cb..92c87f28f8b2 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u32 seed, size_t len);
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
 void btrfs_csum_final(u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
diff --git a/file-item.c b/file-item.c
index 3bf48c68913d..5f6548e9a74f 100644
--- a/file-item.c
+++ b/file-item.c
@@ -312,7 +312,7 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(data, csum_result, len);
+	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
 	btrfs_csum_final(csum_result, (u8 *)&csum_result);
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
-- 
2.16.4

[RFC PATCH 12/17] btrfs-progs: pass checksum type to btrfs_csum_data()

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 check/main.c                |  6 ++++--
 cmds/inspect-dump-super.c   |  8 ++++----
 cmds/rescue-chunk-recover.c | 10 ++++++----
 convert/common.c            |  5 +++--
 disk-io.c                   | 39 +++++++++++++++++++++++++++------------
 disk-io.h                   |  4 ++--
 file-item.c                 |  7 +++++--
 free-space-cache.c          |  2 +-
 image/main.c                |  2 +-
 9 files changed, 53 insertions(+), 30 deletions(-)

diff --git a/check/main.c b/check/main.c
index 92a7e40c1ebf..eabe328b6800 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 	struct btrfs_fs_info *fs_info = root->fs_info;
 	u64 offset = 0;
 	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
 	char *data;
 	unsigned long csum_offset;
 	u32 csum;
@@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 				csum = ~(u32)0;
 				tmp = offset + data_checked;
 
-				csum = btrfs_csum_data((char *)data + tmp,
+				csum = btrfs_csum_data(csum_type,
+						       (char *)data + tmp,
 						(u8 *)&csum, fs_info->sectorsize);
-				btrfs_csum_final(csum, (u8 *)&csum);
+				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
 
 				csum_offset = leaf_offset +
 					 tmp / fs_info->sectorsize * csum_size;
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 96ad3deca3d8..40019a6670ef 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -35,15 +35,15 @@
 #include "kernel-lib/crc32c.h"
 #include "common/help.h"
 
-static int check_csum_sblock(void *sb, int csum_size)
+static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 crc = ~(u32)0;
 
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
 				(u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	return !memcmp(sb, &result, csum_size);
 }
@@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
 	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
 		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
-	else if (check_csum_sblock(sb, csum_size))
+	else if (check_csum_sblock(sb, csum_size, csum_type))
 		printf(" [match]");
 	else
 		printf(" [DON'T MATCH]");
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 0fddb5dd8ead..cddae471f50c 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
 	return dev_offset + data_offset;
 }
 
-static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
+static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
+			  u16 csum_type)
 {
 	char *data;
 	int ret = 0;
@@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_result, (u8 *)&csum_result);
+	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
 	if (csum_result != tree_csum)
 		ret = 1;
 out:
@@ -2102,7 +2103,8 @@ next_csum:
 						  devext->objectid, 1));
 
 		ret = check_one_csum(dev->fd, data_offset, blocksize,
-				     tree_csum);
+				     tree_csum,
+				     btrfs_super_csum_type(root->fs_info->super_copy));
 		if (ret < 0)
 			goto fail_out;
 		else if (ret > 0)
diff --git a/convert/common.c b/convert/common.c
index ab8e6b9f4749..894a6ee0ba90 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
                                   u64 sb_bytenr)
 {
        u32 crc = ~(u32)0;
+       u16 csum_type = btrfs_super_csum_type(sb);
        int ret;
 
-       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
 			     (u8 *)&crc,
                              BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-       btrfs_csum_final(crc, &sb->csum[0]);
+       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
        if (ret < BTRFS_SUPER_INFO_SIZE)
                ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index a0c37c569d58..7e538969c57a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
 {
-	return crc32c(*(u32*)seed, data, len);
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+		return crc32c(*(u32*)seed, data, len);
+	default: /* Not reached */
+		return ~(u32)0;
+	}
+
 }
 
-void btrfs_csum_final(u32 crc, u8 *result)
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
 {
-	put_unaligned_le32(~crc, result);
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+		put_unaligned_le32(~crc, result);
+		break;
+	default: /* Not reached */
+		break;
+	}
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
-	btrfs_csum_final(crc, result);
+	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc, len);
+	btrfs_csum_final(csum_type, crc, result);
 
 	if (verify) {
 		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
@@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	csum_size = btrfs_csum_sizes[csum_type];
 
 	crc = ~(u32)0;
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc,
 			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	if (memcmp(result, sb->csum, csum_size)) {
 		error("superblock checksum mismatch");
@@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	u64 bytenr;
 	u32 crc;
 	int i, ret;
+	u16 csum_type = btrfs_super_csum_type(sb);
 
 	/*
 	 * We need to write super block after all metadata written.
@@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(crc, &sb->csum[0]);
+		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 		btrfs_set_super_bytenr(sb, bytenr);
 
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(crc, &sb->csum[0]);
+		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 92c87f28f8b2..4b5e9ea86385 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u32 crc, u8 *result);
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 5f6548e9a74f..78fdcecd0bab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
 	u16 csum_size =
 		btrfs_super_csum_size(root->fs_info->super_copy);
 
+	u16 csum_type =
+		btrfs_super_csum_type(root->fs_info->super_copy);
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
@@ -312,8 +315,8 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_result, (u8 *)&csum_result);
+	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
 		       (unsigned long long)bytenr);
diff --git a/free-space-cache.c b/free-space-cache.c
index e872eb6a00db..8a57f86dc650 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
 	io_ctl_map_page(io_ctl, 0);
 	crc = crc32c(crc, io_ctl->orig + offset,
 			io_ctl->root->fs_info->sectorsize - offset);
-	btrfs_csum_final(crc, (u8 *)&crc);
+	put_unaligned_le32(~crc, (u8 *)&crc);
 	if (val != crc) {
 		printk("btrfs: csum mismatch on free space cache\n");
 		io_ctl_unmap_page(io_ctl);
diff --git a/image/main.c b/image/main.c
index 28ef1609b5ff..0c8ffede56f5 100644
--- a/image/main.c
+++ b/image/main.c
@@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
 	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
 	u32 crc = ~(u32)0;
 	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	put_unaligned_le32(~crc, result);
 	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
 }
 
-- 
2.16.4

Re: [RFC PATCH 12/17] btrfs-progs: pass checksum type to btrfs_csum_data()

[Nikolay Borisov]

On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:

So patches 12/13 have identical titles, different contents and
absolutely no commit messages :( . How about no... Squash 13 into 12 and
put a commit explaining your change. And by the way the patch's title
doesn't correspond to the content of the patches - you also change
btrfs_checksum_final to take a csum_type


Something as straightforward as the below example should suffice:

"In preparation to supporting new checksum algorithm pass the checksum
type to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate
any differences in processing into the respective functions".

> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  check/main.c                |  6 ++++--
>  cmds/inspect-dump-super.c   |  8 ++++----
>  cmds/rescue-chunk-recover.c | 10 ++++++----
>  convert/common.c            |  5 +++--
>  disk-io.c                   | 39 +++++++++++++++++++++++++++------------
>  disk-io.h                   |  4 ++--
>  file-item.c                 |  7 +++++--
>  free-space-cache.c          |  2 +-
>  image/main.c                |  2 +-
>  9 files changed, 53 insertions(+), 30 deletions(-)
> 
> diff --git a/check/main.c b/check/main.c
> index 92a7e40c1ebf..eabe328b6800 100644
> --- a/check/main.c
> +++ b/check/main.c
> @@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
>  	struct btrfs_fs_info *fs_info = root->fs_info;
>  	u64 offset = 0;
>  	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
>  	char *data;
>  	unsigned long csum_offset;
>  	u32 csum;
> @@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
>  				csum = ~(u32)0;
>  				tmp = offset + data_checked;
>  
> -				csum = btrfs_csum_data((char *)data + tmp,
> +				csum = btrfs_csum_data(csum_type,
> +						       (char *)data + tmp,
>  						(u8 *)&csum, fs_info->sectorsize);
> -				btrfs_csum_final(csum, (u8 *)&csum);
> +				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
>  
>  				csum_offset = leaf_offset +
>  					 tmp / fs_info->sectorsize * csum_size;
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 96ad3deca3d8..40019a6670ef 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -35,15 +35,15 @@
>  #include "kernel-lib/crc32c.h"
>  #include "common/help.h"
>  
> -static int check_csum_sblock(void *sb, int csum_size)
> +static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
>  {
>  	u8 result[BTRFS_CSUM_SIZE];
>  	u32 crc = ~(u32)0;
>  
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
>  				(u8 *)&crc,
>  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	return !memcmp(sb, &result, csum_size);
>  }
> @@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>  	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
>  	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
>  		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> -	else if (check_csum_sblock(sb, csum_size))
> +	else if (check_csum_sblock(sb, csum_size, csum_type))
>  		printf(" [match]");
>  	else
>  		printf(" [DON'T MATCH]");
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 0fddb5dd8ead..cddae471f50c 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
>  	return dev_offset + data_offset;
>  }
>  
> -static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
> +static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
> +			  u16 csum_type)
>  {
>  	char *data;
>  	int ret = 0;
> @@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
>  		goto out;
>  	}
>  	ret = 0;
> -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
>  	if (csum_result != tree_csum)
>  		ret = 1;
>  out:
> @@ -2102,7 +2103,8 @@ next_csum:
>  						  devext->objectid, 1));
>  
>  		ret = check_one_csum(dev->fd, data_offset, blocksize,
> -				     tree_csum);
> +				     tree_csum,
> +				     btrfs_super_csum_type(root->fs_info->super_copy));
>  		if (ret < 0)
>  			goto fail_out;
>  		else if (ret > 0)
> diff --git a/convert/common.c b/convert/common.c
> index ab8e6b9f4749..894a6ee0ba90 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
>                                    u64 sb_bytenr)
>  {
>         u32 crc = ~(u32)0;
> +       u16 csum_type = btrfs_super_csum_type(sb);
>         int ret;
>  
> -       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
>  			     (u8 *)&crc,
>                               BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -       btrfs_csum_final(crc, &sb->csum[0]);
> +       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>         ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
>         if (ret < BTRFS_SUPER_INFO_SIZE)
>                 ret = (ret < 0 ? -errno : -EIO);
> diff --git a/disk-io.c b/disk-io.c
> index a0c37c569d58..7e538969c57a 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
>  	}
>  }
>  
> -u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
> +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
>  {
> -	return crc32c(*(u32*)seed, data, len);
> +	switch (csum_type) {
> +	case BTRFS_CSUM_TYPE_CRC32:
> +		return crc32c(*(u32*)seed, data, len);
> +	default: /* Not reached */
> +		return ~(u32)0;
> +	}
> +
>  }
>  
> -void btrfs_csum_final(u32 crc, u8 *result)
> +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
>  {
> -	put_unaligned_le32(~crc, result);
> +	switch (csum_type) {
> +	case BTRFS_CSUM_TYPE_CRC32:
> +		put_unaligned_le32(~crc, result);
> +		break;
> +	default: /* Not reached */
> +		break;
> +	}
>  }
>  
>  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> @@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>  	u32 crc = ~(u32)0;
>  
>  	len = buf->len - BTRFS_CSUM_SIZE;
> -	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
> -	btrfs_csum_final(crc, result);
> +	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
> +			      (u8 *)&crc, len);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	if (verify) {
>  		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
> @@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
>  	csum_size = btrfs_csum_sizes[csum_type];
>  
>  	crc = ~(u32)0;
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
> +			      (u8 *)&crc,
>  			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	if (memcmp(result, sb->csum, csum_size)) {
>  		error("superblock checksum mismatch");
> @@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  	u64 bytenr;
>  	u32 crc;
>  	int i, ret;
> +	u16 csum_type = btrfs_super_csum_type(sb);
>  
>  	/*
>  	 * We need to write super block after all metadata written.
> @@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
>  		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -		btrfs_csum_final(crc, &sb->csum[0]);
> +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>  
>  		/*
>  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> @@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  		btrfs_set_super_bytenr(sb, bytenr);
>  
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -		btrfs_csum_final(crc, &sb->csum[0]);
> +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>  
>  		/*
>  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> diff --git a/disk-io.h b/disk-io.h
> index 92c87f28f8b2..4b5e9ea86385 100644
> --- a/disk-io.h
> +++ b/disk-io.h
> @@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
>  void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
>  int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
>  int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
> -u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
> -void btrfs_csum_final(u32 crc, u8 *result);
> +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
> +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
>  
>  int btrfs_open_device(struct btrfs_device *dev);
>  int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
> diff --git a/file-item.c b/file-item.c
> index 5f6548e9a74f..78fdcecd0bab 100644
> --- a/file-item.c
> +++ b/file-item.c
> @@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
>  	u16 csum_size =
>  		btrfs_super_csum_size(root->fs_info->super_copy);
>  
> +	u16 csum_type =
> +		btrfs_super_csum_type(root->fs_info->super_copy);
> +
>  	path = btrfs_alloc_path();
>  	if (!path)
>  		return -ENOMEM;
> @@ -312,8 +315,8 @@ csum:
>  	item = (struct btrfs_csum_item *)((unsigned char *)item +
>  					  csum_offset * csum_size);
>  found:
> -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
>  	if (csum_result == 0) {
>  		printk("csum result is 0 for block %llu\n",
>  		       (unsigned long long)bytenr);
> diff --git a/free-space-cache.c b/free-space-cache.c
> index e872eb6a00db..8a57f86dc650 100644
> --- a/free-space-cache.c
> +++ b/free-space-cache.c
> @@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
>  	io_ctl_map_page(io_ctl, 0);
>  	crc = crc32c(crc, io_ctl->orig + offset,
>  			io_ctl->root->fs_info->sectorsize - offset);
> -	btrfs_csum_final(crc, (u8 *)&crc);
> +	put_unaligned_le32(~crc, (u8 *)&crc);
>  	if (val != crc) {
>  		printk("btrfs: csum mismatch on free space cache\n");
>  		io_ctl_unmap_page(io_ctl);
> diff --git a/image/main.c b/image/main.c
> index 28ef1609b5ff..0c8ffede56f5 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
>  	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
>  	u32 crc = ~(u32)0;
>  	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	put_unaligned_le32(~crc, result);
>  	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
>  }
>  
> 

Re: [RFC PATCH 12/17] btrfs-progs: pass checksum type to btrfs_csum_data()

[Johannes Thumshirn]

On Mon, Aug 12, 2019 at 01:51:01PM +0300, Nikolay Borisov wrote:
> 
> 
> On 25.07.19 г. 12:33 ч., Johannes Thumshirn wrote:
> 
> So patches 12/13 have identical titles, different contents and
> absolutely no commit messages :( . How about no... Squash 13 into 12 and
> put a commit explaining your change. And by the way the patch's title
> doesn't correspond to the content of the patches - you also change
> btrfs_checksum_final to take a csum_type

Yep already done.

> 
> 
> Something as straightforward as the below example should suffice:
> 
> "In preparation to supporting new checksum algorithm pass the checksum
> type to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate
> any differences in processing into the respective functions".
> 

Sounds good, will update it.

> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > ---
> >  check/main.c                |  6 ++++--
> >  cmds/inspect-dump-super.c   |  8 ++++----
> >  cmds/rescue-chunk-recover.c | 10 ++++++----
> >  convert/common.c            |  5 +++--
> >  disk-io.c                   | 39 +++++++++++++++++++++++++++------------
> >  disk-io.h                   |  4 ++--
> >  file-item.c                 |  7 +++++--
> >  free-space-cache.c          |  2 +-
> >  image/main.c                |  2 +-
> >  9 files changed, 53 insertions(+), 30 deletions(-)
> > 
> > diff --git a/check/main.c b/check/main.c
> > index 92a7e40c1ebf..eabe328b6800 100644
> > --- a/check/main.c
> > +++ b/check/main.c
> > @@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
> >  	struct btrfs_fs_info *fs_info = root->fs_info;
> >  	u64 offset = 0;
> >  	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
> > +	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
> >  	char *data;
> >  	unsigned long csum_offset;
> >  	u32 csum;
> > @@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
> >  				csum = ~(u32)0;
> >  				tmp = offset + data_checked;
> >  
> > -				csum = btrfs_csum_data((char *)data + tmp,
> > +				csum = btrfs_csum_data(csum_type,
> > +						       (char *)data + tmp,
> >  						(u8 *)&csum, fs_info->sectorsize);
> > -				btrfs_csum_final(csum, (u8 *)&csum);
> > +				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
> >  
> >  				csum_offset = leaf_offset +
> >  					 tmp / fs_info->sectorsize * csum_size;
> > diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> > index 96ad3deca3d8..40019a6670ef 100644
> > --- a/cmds/inspect-dump-super.c
> > +++ b/cmds/inspect-dump-super.c
> > @@ -35,15 +35,15 @@
> >  #include "kernel-lib/crc32c.h"
> >  #include "common/help.h"
> >  
> > -static int check_csum_sblock(void *sb, int csum_size)
> > +static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
> >  {
> >  	u8 result[BTRFS_CSUM_SIZE];
> >  	u32 crc = ~(u32)0;
> >  
> > -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> > +	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
> >  				(u8 *)&crc,
> >  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> > -	btrfs_csum_final(crc, result);
> > +	btrfs_csum_final(csum_type, crc, result);
> >  
> >  	return !memcmp(sb, &result, csum_size);
> >  }
> > @@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> >  	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> >  	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> >  		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> > -	else if (check_csum_sblock(sb, csum_size))
> > +	else if (check_csum_sblock(sb, csum_size, csum_type))
> >  		printf(" [match]");
> >  	else
> >  		printf(" [DON'T MATCH]");
> > diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> > index 0fddb5dd8ead..cddae471f50c 100644
> > --- a/cmds/rescue-chunk-recover.c
> > +++ b/cmds/rescue-chunk-recover.c
> > @@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
> >  	return dev_offset + data_offset;
> >  }
> >  
> > -static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
> > +static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
> > +			  u16 csum_type)
> >  {
> >  	char *data;
> >  	int ret = 0;
> > @@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
> >  		goto out;
> >  	}
> >  	ret = 0;
> > -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> > -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> > +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> > +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
> >  	if (csum_result != tree_csum)
> >  		ret = 1;
> >  out:
> > @@ -2102,7 +2103,8 @@ next_csum:
> >  						  devext->objectid, 1));
> >  
> >  		ret = check_one_csum(dev->fd, data_offset, blocksize,
> > -				     tree_csum);
> > +				     tree_csum,
> > +				     btrfs_super_csum_type(root->fs_info->super_copy));
> >  		if (ret < 0)
> >  			goto fail_out;
> >  		else if (ret > 0)
> > diff --git a/convert/common.c b/convert/common.c
> > index ab8e6b9f4749..894a6ee0ba90 100644
> > --- a/convert/common.c
> > +++ b/convert/common.c
> > @@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
> >                                    u64 sb_bytenr)
> >  {
> >         u32 crc = ~(u32)0;
> > +       u16 csum_type = btrfs_super_csum_type(sb);
> >         int ret;
> >  
> > -       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> > +       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
> >  			     (u8 *)&crc,
> >                               BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> > -       btrfs_csum_final(crc, &sb->csum[0]);
> > +       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
> >         ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
> >         if (ret < BTRFS_SUPER_INFO_SIZE)
> >                 ret = (ret < 0 ? -errno : -EIO);
> > diff --git a/disk-io.c b/disk-io.c
> > index a0c37c569d58..7e538969c57a 100644
> > --- a/disk-io.c
> > +++ b/disk-io.c
> > @@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
> >  	}
> >  }
> >  
> > -u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
> > +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
> >  {
> > -	return crc32c(*(u32*)seed, data, len);
> > +	switch (csum_type) {
> > +	case BTRFS_CSUM_TYPE_CRC32:
> > +		return crc32c(*(u32*)seed, data, len);
> > +	default: /* Not reached */
> > +		return ~(u32)0;
> > +	}
> > +
> >  }
> >  
> > -void btrfs_csum_final(u32 crc, u8 *result)
> > +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
> >  {
> > -	put_unaligned_le32(~crc, result);
> > +	switch (csum_type) {
> > +	case BTRFS_CSUM_TYPE_CRC32:
> > +		put_unaligned_le32(~crc, result);
> > +		break;
> > +	default: /* Not reached */
> > +		break;
> > +	}
> >  }
> >  
> >  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> > @@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> >  	u32 crc = ~(u32)0;
> >  
> >  	len = buf->len - BTRFS_CSUM_SIZE;
> > -	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
> > -	btrfs_csum_final(crc, result);
> > +	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
> > +			      (u8 *)&crc, len);
> > +	btrfs_csum_final(csum_type, crc, result);
> >  
> >  	if (verify) {
> >  		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
> > @@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
> >  	csum_size = btrfs_csum_sizes[csum_type];
> >  
> >  	crc = ~(u32)0;
> > -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> > +	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
> > +			      (u8 *)&crc,
> >  			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> > -	btrfs_csum_final(crc, result);
> > +	btrfs_csum_final(csum_type, crc, result);
> >  
> >  	if (memcmp(result, sb->csum, csum_size)) {
> >  		error("superblock checksum mismatch");
> > @@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
> >  	u64 bytenr;
> >  	u32 crc;
> >  	int i, ret;
> > +	u16 csum_type = btrfs_super_csum_type(sb);
> >  
> >  	/*
> >  	 * We need to write super block after all metadata written.
> > @@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
> >  	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
> >  		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
> >  		crc = ~(u32)0;
> > -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> > +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> >  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> > -		btrfs_csum_final(crc, &sb->csum[0]);
> > +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
> >  
> >  		/*
> >  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> > @@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
> >  		btrfs_set_super_bytenr(sb, bytenr);
> >  
> >  		crc = ~(u32)0;
> > -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> > +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> >  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> > -		btrfs_csum_final(crc, &sb->csum[0]);
> > +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
> >  
> >  		/*
> >  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> > diff --git a/disk-io.h b/disk-io.h
> > index 92c87f28f8b2..4b5e9ea86385 100644
> > --- a/disk-io.h
> > +++ b/disk-io.h
> > @@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
> >  void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
> >  int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
> >  int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
> > -u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
> > -void btrfs_csum_final(u32 crc, u8 *result);
> > +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
> > +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
> >  
> >  int btrfs_open_device(struct btrfs_device *dev);
> >  int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
> > diff --git a/file-item.c b/file-item.c
> > index 5f6548e9a74f..78fdcecd0bab 100644
> > --- a/file-item.c
> > +++ b/file-item.c
> > @@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
> >  	u16 csum_size =
> >  		btrfs_super_csum_size(root->fs_info->super_copy);
> >  
> > +	u16 csum_type =
> > +		btrfs_super_csum_type(root->fs_info->super_copy);
> > +
> >  	path = btrfs_alloc_path();
> >  	if (!path)
> >  		return -ENOMEM;
> > @@ -312,8 +315,8 @@ csum:
> >  	item = (struct btrfs_csum_item *)((unsigned char *)item +
> >  					  csum_offset * csum_size);
> >  found:
> > -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> > -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> > +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> > +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
> >  	if (csum_result == 0) {
> >  		printk("csum result is 0 for block %llu\n",
> >  		       (unsigned long long)bytenr);
> > diff --git a/free-space-cache.c b/free-space-cache.c
> > index e872eb6a00db..8a57f86dc650 100644
> > --- a/free-space-cache.c
> > +++ b/free-space-cache.c
> > @@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
> >  	io_ctl_map_page(io_ctl, 0);
> >  	crc = crc32c(crc, io_ctl->orig + offset,
> >  			io_ctl->root->fs_info->sectorsize - offset);
> > -	btrfs_csum_final(crc, (u8 *)&crc);
> > +	put_unaligned_le32(~crc, (u8 *)&crc);
> >  	if (val != crc) {
> >  		printk("btrfs: csum mismatch on free space cache\n");
> >  		io_ctl_unmap_page(io_ctl);
> > diff --git a/image/main.c b/image/main.c
> > index 28ef1609b5ff..0c8ffede56f5 100644
> > --- a/image/main.c
> > +++ b/image/main.c
> > @@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
> >  	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
> >  	u32 crc = ~(u32)0;
> >  	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> > -	btrfs_csum_final(crc, result);
> > +	put_unaligned_le32(~crc, result);
> >  	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
> >  }
> >  
> > 

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[RFC PATCH 13/17] btrfs-progs: pass checksum type to btrfs_csum_data()

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 932c2a0432ef..d9630f187d0f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb)
 {
 	u8 result[csum_size];
 	u32 crc = ~(u32)0;
+	u16 csum_type = btrfs_super_csum_type(sb);
 
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+	crc = btrfs_csum_data(csum_type,
+			      (char *)sb + BTRFS_CSUM_SIZE,
 				(u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	return !memcmp(sb, &result, csum_size);
 }
@@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb)
 	u8 result[csum_size];
 	struct btrfs_header *hdr;
 	u32 crc = ~(u32)0;
+	u16 csum_type = btrfs_super_csum_type(block);
 
 	if (is_sb) {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+		crc = btrfs_csum_data(csum_type,
+				      (char *)block + BTRFS_CSUM_SIZE,
 				      (u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	} else {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+		crc = btrfs_csum_data(csum_type,
+				      (char *)block + BTRFS_CSUM_SIZE,
 				      (u8 *)&crc,
 				BLOCKSIZE - BTRFS_CSUM_SIZE);
 	}
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
 	memcpy(&hdr->csum, result, csum_size);
-- 
2.16.4

[RFC PATCH 14/17] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c

[Johannes Thumshirn]

update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb'
argument set to 1.

Get rid of the special case for is_sb == 0.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index d9630f187d0f..9e64b34d2b8f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb)
 	return !memcmp(sb, &result, csum_size);
 }
 
-static void update_block_csum(void *block, int is_sb)
+static void update_block_csum(void *block)
 {
 	u8 result[csum_size];
 	struct btrfs_header *hdr;
 	u32 crc = ~(u32)0;
 	u16 csum_type = btrfs_super_csum_type(block);
 
-	if (is_sb) {
-		crc = btrfs_csum_data(csum_type,
-				      (char *)block + BTRFS_CSUM_SIZE,
-				      (u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	} else {
-		crc = btrfs_csum_data(csum_type,
-				      (char *)block + BTRFS_CSUM_SIZE,
-				      (u8 *)&crc,
-				BLOCKSIZE - BTRFS_CSUM_SIZE);
-	}
+	crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc,
+			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+
 	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
@@ -354,7 +347,7 @@ int main(int argc, char **argv) {
 
 	if (changed) {
 		printf("Update csum\n");
-		update_block_csum(buf, 1);
+		update_block_csum(buf);
 		ret = pwrite(fd, buf, BLOCKSIZE, off);
 		if (ret <= 0) {
 			printf("pwrite error %d at offset %llu\n",
-- 
2.16.4

Re: [RFC PATCH 14/17] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c

[Nikolay Borisov]

On 25.07.19 г. 12:34 ч., Johannes Thumshirn wrote:
> update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb'
> argument set to 1.
> 
> Get rid of the special case for is_sb == 0.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  btrfs-sb-mod.c | 19 ++++++-------------
>  1 file changed, 6 insertions(+), 13 deletions(-)
> 
> diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
> index d9630f187d0f..9e64b34d2b8f 100644
> --- a/btrfs-sb-mod.c
> +++ b/btrfs-sb-mod.c
> @@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb)
>  	return !memcmp(sb, &result, csum_size);
>  }
>  
> -static void update_block_csum(void *block, int is_sb)
> +static void update_block_csum(void *block)
>  {
>  	u8 result[csum_size];
>  	struct btrfs_header *hdr;
>  	u32 crc = ~(u32)0;
>  	u16 csum_type = btrfs_super_csum_type(block);
>  
> -	if (is_sb) {
> -		crc = btrfs_csum_data(csum_type,
> -				      (char *)block + BTRFS_CSUM_SIZE,
> -				      (u8 *)&crc,
> -				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	} else {
> -		crc = btrfs_csum_data(csum_type,
> -				      (char *)block + BTRFS_CSUM_SIZE,
> -				      (u8 *)&crc,
> -				BLOCKSIZE - BTRFS_CSUM_SIZE);
> -	}
> +	crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
> +			      (u8 *)&crc,
> +			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> +
>  	btrfs_csum_final(csum_type, crc, result);
>  	memset(block, 0, BTRFS_CSUM_SIZE);
>  	hdr = (struct btrfs_header *)block;
> @@ -354,7 +347,7 @@ int main(int argc, char **argv) {
>  
>  	if (changed) {
>  		printf("Update csum\n");
> -		update_block_csum(buf, 1);
> +		update_block_csum(buf);
>  		ret = pwrite(fd, buf, BLOCKSIZE, off);
>  		if (ret <= 0) {
>  			printf("pwrite error %d at offset %llu\n",
> 

[RFC PATCH 15/17] btrfs-progs: update checksumming api

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

[XXX Not done!!! ]

This is based on a patch by David Sterba.

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c              | 21 ++++++----------
 check/main.c                | 20 +++++++--------
 cmds/inspect-dump-super.c   |  9 +++----
 cmds/rescue-chunk-recover.c | 12 ++++++---
 convert/common.c            |  9 +++----
 disk-io.c                   | 59 ++++++++++++++++++---------------------------
 disk-io.h                   |  3 +--
 file-item.c                 |  8 +++---
 8 files changed, 60 insertions(+), 81 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 9e64b34d2b8f..105b556b0cf1 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -33,31 +33,24 @@ static int csum_size;
 
 static int check_csum_superblock(void *sb)
 {
-	u8 result[csum_size];
-	u32 crc = ~(u32)0;
+	u8 result[BTRFS_CSUM_SIZE];
 	u16 csum_type = btrfs_super_csum_type(sb);
 
-	crc = btrfs_csum_data(csum_type,
-			      (char *)sb + BTRFS_CSUM_SIZE,
-				(u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	return !memcmp(sb, &result, csum_size);
+	return !memcmp(sb, result, csum_size);
 }
 
 static void update_block_csum(void *block)
 {
-	u8 result[csum_size];
+	u8 result[BTRFS_CSUM_SIZE];
 	struct btrfs_header *hdr;
-	u32 crc = ~(u32)0;
 	u16 csum_type = btrfs_super_csum_type(block);
 
-	crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc,
-			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+	btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
 	memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index eabe328b6800..429edc4aa735 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5582,10 +5582,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 	u64 offset = 0;
 	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
 	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
-	char *data;
+	u8 *data;
 	unsigned long csum_offset;
-	u32 csum;
-	u32 csum_expected;
+	u8 result[BTRFS_CSUM_SIZE];
+	u8 csum_expected[BTRFS_CSUM_SIZE];
 	u64 read_len;
 	u64 data_checked = 0;
 	u64 tmp;
@@ -5611,7 +5611,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 		for (mirror = 1; mirror <= num_copies; mirror++) {
 			read_len = num_bytes - offset;
 			/* read as much space once a time */
-			ret = read_extent_data(fs_info, data + offset,
+			ret = read_extent_data(fs_info, (char *)data + offset,
 					bytenr + offset, &read_len, mirror);
 			if (ret)
 				goto out;
@@ -5619,24 +5619,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 			data_checked = 0;
 			/* verify every 4k data's checksum */
 			while (data_checked < read_len) {
-				csum = ~(u32)0;
 				tmp = offset + data_checked;
 
-				csum = btrfs_csum_data(csum_type,
-						       (char *)data + tmp,
-						(u8 *)&csum, fs_info->sectorsize);
-				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
+				btrfs_csum_data(csum_type, data + tmp,
+						result, fs_info->sectorsize);
 
 				csum_offset = leaf_offset +
 					 tmp / fs_info->sectorsize * csum_size;
 				read_extent_buffer(eb, (char *)&csum_expected,
 						   csum_offset, csum_size);
-				if (csum != csum_expected) {
+				if (memcmp(result, csum_expected, csum_size) != 0) {
 					csum_mismatch = true;
+					/* FIXME: format */
 					fprintf(stderr,
 			"mirror %d bytenr %llu csum %u expected csum %u\n",
 						mirror, bytenr + tmp,
-						csum, csum_expected);
+						result[0], csum_expected[0]);
 				}
 				data_checked += fs_info->sectorsize;
 			}
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 40019a6670ef..58bf82b0bbd3 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -38,14 +38,11 @@
 static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
-	u32 crc = ~(u32)0;
 
-	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
-				(u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	return !memcmp(sb, &result, csum_size);
+	return !memcmp(sb, result, csum_size);
 }
 
 static void print_sys_chunk_array(struct btrfs_super_block *sb)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index cddae471f50c..4b6426ae0346 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
 {
 	char *data;
 	int ret = 0;
-	u32 csum_result = ~(u32)0;
+	u8 result[BTRFS_CSUM_SIZE];
+	int csum_size = 0;
+	u8 expected_csum[BTRFS_CSUM_SIZE];
+
+	ASSERT(0);
 
 	data = malloc(len);
 	if (!data)
@@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
-	if (csum_result != tree_csum)
+	put_unaligned_le32(tree_csum, expected_csum);
+	btrfs_csum_data(csum_type, (u8 *)data, result, len);
+	if (memcmp(result, expected_csum, csum_size) != 0)
 		ret = 1;
 out:
 	free(data);
diff --git a/convert/common.c b/convert/common.c
index 894a6ee0ba90..8cae507ec0f7 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
 static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
                                   u64 sb_bytenr)
 {
-       u32 crc = ~(u32)0;
+       u8 result[BTRFS_CSUM_SIZE];
        u16 csum_type = btrfs_super_csum_type(sb);
        int ret;
 
-       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
-			     (u8 *)&crc,
-                             BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+       btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+		       result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+       memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
        if (ret < BTRFS_SUPER_INFO_SIZE)
                ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index 7e538969c57a..810c2e14294a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 {
-	switch (csum_type) {
-	case BTRFS_CSUM_TYPE_CRC32:
-		return crc32c(*(u32*)seed, data, len);
-	default: /* Not reached */
-		return ~(u32)0;
-	}
-
-}
+	u32 crc = ~(u32)0;
+	memset(out, 0, BTRFS_CSUM_SIZE);
 
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
-{
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
-		put_unaligned_le32(~crc, result);
-		break;
-	default: /* Not reached */
-		break;
+		crc = crc32c(crc, data, len);
+		put_unaligned_le32(~crc, out);
+		return 0;
+	default:
+		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
+		ASSERT(0);
 	}
+
+	return -1;
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 len;
-	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc, len);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE,
+			result, len);
 
 	if (verify) {
 		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
+			/* FIXME: format */
 			if (!silent)
 				printk("checksum verify failed on %llu found %08X wanted %08X\n",
 				       (unsigned long long)buf->start,
-				       *((u32 *)result),
-				       *((u32*)(char *)buf->data));
+				       result[0],
+				       buf->data[0]);
 			return 1;
 		}
 	} else {
@@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 {
 	u8 result[BTRFS_CSUM_SIZE];
-	u32 crc;
 	u16 csum_type;
 	int csum_size;
 	u8 *metadata_uuid;
@@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	}
 	csum_size = btrfs_csum_sizes[csum_type];
 
-	crc = ~(u32)0;
-	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc,
-			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
 	if (memcmp(result, sb->csum, csum_size)) {
 		error("superblock checksum mismatch");
@@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 			    struct btrfs_device *device)
 {
 	u64 bytenr;
-	u32 crc;
+	u8 result[BTRFS_CSUM_SIZE];
 	int i, ret;
 	u16 csum_type = btrfs_super_csum_type(sb);
 
@@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	}
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
-		crc = ~(u32)0;
-		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
-				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+		btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+		memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 
 		btrfs_set_super_bytenr(sb, bytenr);
 
-		crc = ~(u32)0;
-		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+		memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 4b5e9ea86385..545cacda9a79 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 78fdcecd0bab..c6e9d212bcab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
 	struct btrfs_csum_item *item;
 	struct extent_buffer *leaf = NULL;
 	u64 csum_offset;
-	u32 csum_result = ~(u32)0;
+	u8 csum_result[BTRFS_CSUM_SIZE];
 	u32 sectorsize = root->fs_info->sectorsize;
 	u32 nritems;
 	u32 ins_size;
@@ -315,14 +315,14 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
+	btrfs_csum_data(csum_type, (u8 *)data, csum_result, len);
+	/* FIXME: does not make sense for non-crc32c */
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
 		       (unsigned long long)bytenr);
 	}
 
-	write_extent_buffer(leaf, &csum_result, (unsigned long)item,
+	write_extent_buffer(leaf, csum_result, (unsigned long)item,
 			    csum_size);
 	btrfs_mark_buffer_dirty(path->nodes[0]);
 fail:
-- 
2.16.4

[RFC PATCH 16/17] btrfs-progs: add xxhash sources

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

git://github.com/Cyan4973/xxHash

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 crypto/xxhash.c | 1024 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 crypto/xxhash.h |  445 ++++++++++++++++++++++++
 2 files changed, 1469 insertions(+)
 create mode 100644 crypto/xxhash.c
 create mode 100644 crypto/xxhash.h

diff --git a/crypto/xxhash.c b/crypto/xxhash.c
new file mode 100644
index 000000000000..af9d02795ac6
--- /dev/null
+++ b/crypto/xxhash.c
@@ -0,0 +1,1024 @@
+/*
+*  xxHash - Fast Hash algorithm
+*  Copyright (C) 2012-2016, Yann Collet
+*
+*  BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+*
+*  Redistribution and use in source and binary forms, with or without
+*  modification, are permitted provided that the following conditions are
+*  met:
+*
+*  * Redistributions of source code must retain the above copyright
+*  notice, this list of conditions and the following disclaimer.
+*  * Redistributions in binary form must reproduce the above
+*  copyright notice, this list of conditions and the following disclaimer
+*  in the documentation and/or other materials provided with the
+*  distribution.
+*
+*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+*  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+*  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+*  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+*  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+*  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+*  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+*  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+*  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*
+*  You can contact the author at :
+*  - xxHash homepage: http://www.xxhash.com
+*  - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+
+/* *************************************
+*  Tuning parameters
+***************************************/
+/*!XXH_FORCE_MEMORY_ACCESS :
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
+ *            This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
+ * Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
+ *            It can generate buggy code on targets which do not support unaligned memory accesses.
+ *            But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See http://stackoverflow.com/a/32095106/646947 for details.
+ * Prefer these methods in priority order (0 > 1 > 2)
+ */
+#ifndef XXH_FORCE_MEMORY_ACCESS   /* can be defined externally, on command line for example */
+#  if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
+                        || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
+                        || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+#    define XXH_FORCE_MEMORY_ACCESS 2
+#  elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \
+  (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
+                    || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
+                    || defined(__ARM_ARCH_7S__) ))
+#    define XXH_FORCE_MEMORY_ACCESS 1
+#  endif
+#endif
+
+/*!XXH_ACCEPT_NULL_INPUT_POINTER :
+ * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault.
+ * When this macro is enabled, xxHash actively checks input for null pointer.
+ * It it is, result for null input pointers is the same as a null-length input.
+ */
+#ifndef XXH_ACCEPT_NULL_INPUT_POINTER   /* can be defined externally */
+#  define XXH_ACCEPT_NULL_INPUT_POINTER 0
+#endif
+
+/*!XXH_FORCE_ALIGN_CHECK :
+ * This is a minor performance trick, only useful with lots of very small keys.
+ * It means : check for aligned/unaligned input.
+ * The check costs one initial branch per hash;
+ * set it to 0 when the input is guaranteed to be aligned,
+ * or when alignment doesn't matter for performance.
+ */
+#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
+#  if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
+#    define XXH_FORCE_ALIGN_CHECK 0
+#  else
+#    define XXH_FORCE_ALIGN_CHECK 1
+#  endif
+#endif
+
+
+/* *************************************
+*  Includes & Memory related functions
+***************************************/
+/*! Modify the local functions below should you wish to use some other memory routines
+*   for malloc(), free() */
+#include <stdlib.h>
+static void* XXH_malloc(size_t s) { return malloc(s); }
+static void  XXH_free  (void* p)  { free(p); }
+/*! and for memcpy() */
+#include <string.h>
+static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
+
+#include <assert.h>   /* assert */
+
+#define XXH_STATIC_LINKING_ONLY
+#include "xxhash.h"
+
+
+/* *************************************
+*  Compiler Specific Options
+***************************************/
+#ifdef _MSC_VER    /* Visual Studio */
+#  pragma warning(disable : 4127)      /* disable: C4127: conditional expression is constant */
+#  define XXH_FORCE_INLINE static __forceinline
+#  define XXH_NO_INLINE static __declspec(noinline)
+#else
+#  if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#    ifdef __GNUC__
+#      define XXH_FORCE_INLINE static inline __attribute__((always_inline))
+#      define XXH_NO_INLINE static __attribute__((noinline))
+#    else
+#      define XXH_FORCE_INLINE static inline
+#      define XXH_NO_INLINE static
+#    endif
+#  else
+#    define XXH_FORCE_INLINE static
+#    define XXH_NO_INLINE static
+#  endif /* __STDC_VERSION__ */
+#endif
+
+
+/* *************************************
+*  Basic Types
+***************************************/
+#ifndef MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint8_t  BYTE;
+    typedef uint16_t U16;
+    typedef uint32_t U32;
+# else
+    typedef unsigned char      BYTE;
+    typedef unsigned short     U16;
+    typedef unsigned int       U32;
+# endif
+#endif
+
+
+/* ===   Memory access   === */
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; } __attribute__((packed)) unalign;
+static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+static U32 XXH_read32(const void* memPtr)
+{
+    U32 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+
+/* ===   Endianess   === */
+typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
+
+/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
+#ifndef XXH_CPU_LITTLE_ENDIAN
+static int XXH_isLittleEndian(void)
+{
+    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
+    return one.c[0];
+}
+#   define XXH_CPU_LITTLE_ENDIAN   XXH_isLittleEndian()
+#endif
+
+
+
+
+/* ****************************************
+*  Compiler-specific Functions and Macros
+******************************************/
+#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
+
+/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
+#if defined(_MSC_VER)
+#  define XXH_rotl32(x,r) _rotl(x,r)
+#  define XXH_rotl64(x,r) _rotl64(x,r)
+#else
+#  define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r))))
+#  define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r))))
+#endif
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap32 _byteswap_ulong
+#elif XXH_GCC_VERSION >= 403
+#  define XXH_swap32 __builtin_bswap32
+#else
+static U32 XXH_swap32 (U32 x)
+{
+    return  ((x << 24) & 0xff000000 ) |
+            ((x <<  8) & 0x00ff0000 ) |
+            ((x >>  8) & 0x0000ff00 ) |
+            ((x >> 24) & 0x000000ff );
+}
+#endif
+
+
+/* ***************************
+*  Memory reads
+*****************************/
+typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
+
+XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
+}
+
+static U32 XXH_readBE32(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
+}
+
+XXH_FORCE_INLINE U32
+XXH_readLE32_align(const void* ptr, XXH_alignment align)
+{
+    if (align==XXH_unaligned) {
+        return XXH_readLE32(ptr);
+    } else {
+        return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
+    }
+}
+
+
+/* *************************************
+*  Macros
+***************************************/
+#define XXH_STATIC_ASSERT(c)  { enum { XXH_sa = 1/(int)(!!(c)) }; }  /* use after variable declarations */
+XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }
+
+
+/* *******************************************************************
+*  32-bit hash functions
+*********************************************************************/
+static const U32 PRIME32_1 = 2654435761U;   /* 0b10011110001101110111100110110001 */
+static const U32 PRIME32_2 = 2246822519U;   /* 0b10000101111010111100101001110111 */
+static const U32 PRIME32_3 = 3266489917U;   /* 0b11000010101100101010111000111101 */
+static const U32 PRIME32_4 =  668265263U;   /* 0b00100111110101001110101100101111 */
+static const U32 PRIME32_5 =  374761393U;   /* 0b00010110010101100110011110110001 */
+
+static U32 XXH32_round(U32 acc, U32 input)
+{
+    acc += input * PRIME32_2;
+    acc  = XXH_rotl32(acc, 13);
+    acc *= PRIME32_1;
+#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE)
+    /* UGLY HACK:
+     * This inline assembly hack forces acc into a normal register. This is the
+     * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop
+     * (pragmas and attributes don't work for some resason) without globally
+     * disabling SSE4.1.
+     *
+     * The reason we want to avoid vectorization is because despite working on
+     * 4 integers at a time, there are multiple factors slowing XXH32 down on
+     * SSE4:
+     * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!)
+     *   making it slightly slower to multiply four integers at once compared to four
+     *   integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is
+     *   still not worth it to go into SSE just to multiply unless doing a long operation.
+     *
+     * - Four instructions are required to rotate,
+     *      movqda tmp,  v // not required with VEX encoding
+     *      pslld  tmp, 13 // tmp <<= 13
+     *      psrld  v,   19 // x >>= 19
+     *      por    v,  tmp // x |= tmp
+     *   compared to one for scalar:
+     *      roll   v, 13    // reliably fast across the board
+     *      shldl  v, v, 13 // Sandy Bridge and later prefer this for some reason
+     *
+     * - Instruction level parallelism is actually more beneficial here because the
+     *   SIMD actually serializes this operation: While v1 is rotating, v2 can load data,
+     *   while v3 can multiply. SSE forces them to operate together.
+     *
+     * How this hack works:
+     * __asm__(""       // Declare an assembly block but don't declare any instructions
+     *          :       // However, as an Input/Output Operand,
+     *          "+r"    // constrain a read/write operand (+) as a general purpose register (r).
+     *          (acc)   // and set acc as the operand
+     * );
+     *
+     * Because of the 'r', the compiler has promised that seed will be in a
+     * general purpose register and the '+' says that it will be 'read/write',
+     * so it has to assume it has changed. It is like volatile without all the
+     * loads and stores.
+     *
+     * Since the argument has to be in a normal register (not an SSE register),
+     * each time XXH32_round is called, it is impossible to vectorize. */
+    __asm__("" : "+r" (acc));
+#endif
+    return acc;
+}
+
+/* mix all bits */
+static U32 XXH32_avalanche(U32 h32)
+{
+    h32 ^= h32 >> 15;
+    h32 *= PRIME32_2;
+    h32 ^= h32 >> 13;
+    h32 *= PRIME32_3;
+    h32 ^= h32 >> 16;
+    return(h32);
+}
+
+#define XXH_get32bits(p) XXH_readLE32_align(p, align)
+
+static U32
+XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align)
+
+{
+    const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1               \
+    h32 += (*p++) * PRIME32_5; \
+    h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
+
+#define PROCESS4                         \
+    h32 += XXH_get32bits(p) * PRIME32_3; \
+    p+=4;                                \
+    h32  = XXH_rotl32(h32, 17) * PRIME32_4 ;
+
+    switch(len&15)  /* or switch(bEnd - p) */
+    {
+      case 12:      PROCESS4;
+                    /* fallthrough */
+      case 8:       PROCESS4;
+                    /* fallthrough */
+      case 4:       PROCESS4;
+                    return XXH32_avalanche(h32);
+
+      case 13:      PROCESS4;
+                    /* fallthrough */
+      case 9:       PROCESS4;
+                    /* fallthrough */
+      case 5:       PROCESS4;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 14:      PROCESS4;
+                    /* fallthrough */
+      case 10:      PROCESS4;
+                    /* fallthrough */
+      case 6:       PROCESS4;
+                    PROCESS1;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 15:      PROCESS4;
+                    /* fallthrough */
+      case 11:      PROCESS4;
+                    /* fallthrough */
+      case 7:       PROCESS4;
+                    /* fallthrough */
+      case 3:       PROCESS1;
+                    /* fallthrough */
+      case 2:       PROCESS1;
+                    /* fallthrough */
+      case 1:       PROCESS1;
+                    /* fallthrough */
+      case 0:       return XXH32_avalanche(h32);
+    }
+    assert(0);
+    return h32;   /* reaching this point is deemed impossible */
+}
+
+XXH_FORCE_INLINE U32
+XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U32 h32;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)16;
+    }
+#endif
+
+    if (len>=16) {
+        const BYTE* const limit = bEnd - 15;
+        U32 v1 = seed + PRIME32_1 + PRIME32_2;
+        U32 v2 = seed + PRIME32_2;
+        U32 v3 = seed + 0;
+        U32 v4 = seed - PRIME32_1;
+
+        do {
+            v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
+            v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
+            v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
+            v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
+        } while (p < limit);
+
+        h32 = XXH_rotl32(v1, 1)  + XXH_rotl32(v2, 7)
+            + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
+    } else {
+        h32  = seed + PRIME32_5;
+    }
+
+    h32 += (U32)len;
+
+    return XXH32_finalize(h32, p, len&15, align);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH32_state_t state;
+    XXH32_reset(&state, seed);
+    XXH32_update(&state, input, len);
+    return XXH32_digest(&state);
+
+#else
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 3) == 0) {   /* Input is 4-bytes aligned, leverage the speed benefit */
+            return XXH32_endian_align(input, len, seed, XXH_aligned);
+    }   }
+
+    return XXH32_endian_align(input, len, seed, XXH_unaligned);
+#endif
+}
+
+
+
+/*======   Hash streaming   ======*/
+
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
+{
+    return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
+{
+    XXH32_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME32_1 + PRIME32_2;
+    state.v2 = seed + PRIME32_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME32_1;
+    /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API XXH_errorcode
+XXH32_update(XXH32_state_t* state, const void* input, size_t len)
+{
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    {   const BYTE* p = (const BYTE*)input;
+        const BYTE* const bEnd = p + len;
+
+        state->total_len_32 += (XXH32_hash_t)len;
+        state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16));
+
+        if (state->memsize + len < 16)  {   /* fill in tmp buffer */
+            XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
+            state->memsize += (XXH32_hash_t)len;
+            return XXH_OK;
+        }
+
+        if (state->memsize) {   /* some data left from previous update */
+            XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
+            {   const U32* p32 = state->mem32;
+                state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++;
+                state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++;
+                state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++;
+                state->v4 = XXH32_round(state->v4, XXH_readLE32(p32));
+            }
+            p += 16-state->memsize;
+            state->memsize = 0;
+        }
+
+        if (p <= bEnd-16) {
+            const BYTE* const limit = bEnd - 16;
+            U32 v1 = state->v1;
+            U32 v2 = state->v2;
+            U32 v3 = state->v3;
+            U32 v4 = state->v4;
+
+            do {
+                v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4;
+                v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4;
+                v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4;
+                v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4;
+            } while (p<=limit);
+
+            state->v1 = v1;
+            state->v2 = v2;
+            state->v3 = v3;
+            state->v4 = v4;
+        }
+
+        if (p < bEnd) {
+            XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
+            state->memsize = (unsigned)(bEnd-p);
+        }
+    }
+
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state)
+{
+    U32 h32;
+
+    if (state->large_len) {
+        h32 = XXH_rotl32(state->v1, 1)
+            + XXH_rotl32(state->v2, 7)
+            + XXH_rotl32(state->v3, 12)
+            + XXH_rotl32(state->v4, 18);
+    } else {
+        h32 = state->v3 /* == seed */ + PRIME32_5;
+    }
+
+    h32 += state->total_len_32;
+
+    return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned);
+}
+
+
+/*======   Canonical representation   ======*/
+
+/*! Default XXH result types are basic unsigned 32 and 64 bits.
+*   The canonical representation follows human-readable write convention, aka big-endian (large digits first).
+*   These functions allow transformation of hash result into and from its canonical format.
+*   This way, hash values can be written into a file or buffer, remaining comparable across different systems.
+*/
+
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
+{
+    return XXH_readBE32(src);
+}
+
+
+#ifndef XXH_NO_LONG_LONG
+
+/* *******************************************************************
+*  64-bit hash functions
+*********************************************************************/
+
+/*======   Memory access   ======*/
+
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint64_t U64;
+# else
+    /* if compiler doesn't support unsigned long long, replace by another 64-bit type */
+    typedef unsigned long long U64;
+# endif
+#endif
+
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64;
+static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+
+static U64 XXH_read64(const void* memPtr)
+{
+    U64 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap64 _byteswap_uint64
+#elif XXH_GCC_VERSION >= 403
+#  define XXH_swap64 __builtin_bswap64
+#else
+static U64 XXH_swap64 (U64 x)
+{
+    return  ((x << 56) & 0xff00000000000000ULL) |
+            ((x << 40) & 0x00ff000000000000ULL) |
+            ((x << 24) & 0x0000ff0000000000ULL) |
+            ((x << 8)  & 0x000000ff00000000ULL) |
+            ((x >> 8)  & 0x00000000ff000000ULL) |
+            ((x >> 24) & 0x0000000000ff0000ULL) |
+            ((x >> 40) & 0x000000000000ff00ULL) |
+            ((x >> 56) & 0x00000000000000ffULL);
+}
+#endif
+
+XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
+}
+
+static U64 XXH_readBE64(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
+}
+
+XXH_FORCE_INLINE U64
+XXH_readLE64_align(const void* ptr, XXH_alignment align)
+{
+    if (align==XXH_unaligned)
+        return XXH_readLE64(ptr);
+    else
+        return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
+}
+
+
+/*======   xxh64   ======*/
+
+static const U64 PRIME64_1 = 11400714785074694791ULL;   /* 0b1001111000110111011110011011000110000101111010111100101010000111 */
+static const U64 PRIME64_2 = 14029467366897019727ULL;   /* 0b1100001010110010101011100011110100100111110101001110101101001111 */
+static const U64 PRIME64_3 =  1609587929392839161ULL;   /* 0b0001011001010110011001111011000110011110001101110111100111111001 */
+static const U64 PRIME64_4 =  9650029242287828579ULL;   /* 0b1000010111101011110010100111011111000010101100101010111001100011 */
+static const U64 PRIME64_5 =  2870177450012600261ULL;   /* 0b0010011111010100111010110010111100010110010101100110011111000101 */
+
+static U64 XXH64_round(U64 acc, U64 input)
+{
+    acc += input * PRIME64_2;
+    acc  = XXH_rotl64(acc, 31);
+    acc *= PRIME64_1;
+    return acc;
+}
+
+static U64 XXH64_mergeRound(U64 acc, U64 val)
+{
+    val  = XXH64_round(0, val);
+    acc ^= val;
+    acc  = acc * PRIME64_1 + PRIME64_4;
+    return acc;
+}
+
+static U64 XXH64_avalanche(U64 h64)
+{
+    h64 ^= h64 >> 33;
+    h64 *= PRIME64_2;
+    h64 ^= h64 >> 29;
+    h64 *= PRIME64_3;
+    h64 ^= h64 >> 32;
+    return h64;
+}
+
+
+#define XXH_get64bits(p) XXH_readLE64_align(p, align)
+
+static U64
+XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1_64            \
+    h64 ^= (*p++) * PRIME64_5; \
+    h64 = XXH_rotl64(h64, 11) * PRIME64_1;
+
+#define PROCESS4_64          \
+    h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \
+    p+=4;                    \
+    h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
+
+#define PROCESS8_64 {        \
+    U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \
+    p+=8;                    \
+    h64 ^= k1;               \
+    h64  = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \
+}
+
+    switch(len&31) {
+      case 24: PROCESS8_64;
+                    /* fallthrough */
+      case 16: PROCESS8_64;
+                    /* fallthrough */
+      case  8: PROCESS8_64;
+               return XXH64_avalanche(h64);
+
+      case 28: PROCESS8_64;
+                    /* fallthrough */
+      case 20: PROCESS8_64;
+                    /* fallthrough */
+      case 12: PROCESS8_64;
+                    /* fallthrough */
+      case  4: PROCESS4_64;
+               return XXH64_avalanche(h64);
+
+      case 25: PROCESS8_64;
+                    /* fallthrough */
+      case 17: PROCESS8_64;
+                    /* fallthrough */
+      case  9: PROCESS8_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 29: PROCESS8_64;
+                    /* fallthrough */
+      case 21: PROCESS8_64;
+                    /* fallthrough */
+      case 13: PROCESS8_64;
+                    /* fallthrough */
+      case  5: PROCESS4_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 26: PROCESS8_64;
+                    /* fallthrough */
+      case 18: PROCESS8_64;
+                    /* fallthrough */
+      case 10: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 30: PROCESS8_64;
+                    /* fallthrough */
+      case 22: PROCESS8_64;
+                    /* fallthrough */
+      case 14: PROCESS8_64;
+                    /* fallthrough */
+      case  6: PROCESS4_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 27: PROCESS8_64;
+                    /* fallthrough */
+      case 19: PROCESS8_64;
+                    /* fallthrough */
+      case 11: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 31: PROCESS8_64;
+                    /* fallthrough */
+      case 23: PROCESS8_64;
+                    /* fallthrough */
+      case 15: PROCESS8_64;
+                    /* fallthrough */
+      case  7: PROCESS4_64;
+                    /* fallthrough */
+      case  3: PROCESS1_64;
+                    /* fallthrough */
+      case  2: PROCESS1_64;
+                    /* fallthrough */
+      case  1: PROCESS1_64;
+                    /* fallthrough */
+      case  0: return XXH64_avalanche(h64);
+    }
+
+    /* impossible to reach */
+    assert(0);
+    return 0;  /* unreachable, but some compilers complain without it */
+}
+
+XXH_FORCE_INLINE U64
+XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U64 h64;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)32;
+    }
+#endif
+
+    if (len>=32) {
+        const BYTE* const limit = bEnd - 32;
+        U64 v1 = seed + PRIME64_1 + PRIME64_2;
+        U64 v2 = seed + PRIME64_2;
+        U64 v3 = seed + 0;
+        U64 v4 = seed - PRIME64_1;
+
+        do {
+            v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
+            v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
+            v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
+            v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
+        } while (p<=limit);
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+
+    } else {
+        h64  = seed + PRIME64_5;
+    }
+
+    h64 += (U64) len;
+
+    return XXH64_finalize(h64, p, len, align);
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH64_state_t state;
+    XXH64_reset(&state, seed);
+    XXH64_update(&state, input, len);
+    return XXH64_digest(&state);
+
+#else
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 7)==0) {  /* Input is aligned, let's leverage the speed advantage */
+            return XXH64_endian_align(input, len, seed, XXH_aligned);
+    }   }
+
+    return XXH64_endian_align(input, len, seed, XXH_unaligned);
+
+#endif
+}
+
+/*======   Hash Streaming   ======*/
+
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
+{
+    return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
+{
+    XXH64_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME64_1 + PRIME64_2;
+    state.v2 = seed + PRIME64_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME64_1;
+     /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API XXH_errorcode
+XXH64_update (XXH64_state_t* state, const void* input, size_t len)
+{
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    {   const BYTE* p = (const BYTE*)input;
+        const BYTE* const bEnd = p + len;
+
+        state->total_len += len;
+
+        if (state->memsize + len < 32) {  /* fill in tmp buffer */
+            XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+            state->memsize += (U32)len;
+            return XXH_OK;
+        }
+
+        if (state->memsize) {   /* tmp buffer is full */
+            XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
+            state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0));
+            state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1));
+            state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2));
+            state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3));
+            p += 32-state->memsize;
+            state->memsize = 0;
+        }
+
+        if (p+32 <= bEnd) {
+            const BYTE* const limit = bEnd - 32;
+            U64 v1 = state->v1;
+            U64 v2 = state->v2;
+            U64 v3 = state->v3;
+            U64 v4 = state->v4;
+
+            do {
+                v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8;
+                v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8;
+                v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8;
+                v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8;
+            } while (p<=limit);
+
+            state->v1 = v1;
+            state->v2 = v2;
+            state->v3 = v3;
+            state->v4 = v4;
+        }
+
+        if (p < bEnd) {
+            XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
+            state->memsize = (unsigned)(bEnd-p);
+        }
+    }
+
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state)
+{
+    U64 h64;
+
+    if (state->total_len >= 32) {
+        U64 const v1 = state->v1;
+        U64 const v2 = state->v2;
+        U64 const v3 = state->v3;
+        U64 const v4 = state->v4;
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+    } else {
+        h64  = state->v3 /*seed*/ + PRIME64_5;
+    }
+
+    h64 += (U64) state->total_len;
+
+    return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned);
+}
+
+
+/*====== Canonical representation   ======*/
+
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
+{
+    return XXH_readBE64(src);
+}
+
+
+
+/* *********************************************************************
+*  XXH3
+*  New generation hash designed for speed on small keys and vectorization
+************************************************************************ */
+
+#include "xxh3.h"
+
+
+#endif  /* XXH_NO_LONG_LONG */
diff --git a/crypto/xxhash.h b/crypto/xxhash.h
new file mode 100644
index 000000000000..9ee05e5dc146
--- /dev/null
+++ b/crypto/xxhash.h
@@ -0,0 +1,445 @@
+/*
+   xxHash - Extremely Fast Hash algorithm
+   Header File
+   Copyright (C) 2012-2016, Yann Collet.
+
+   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+
+   Redistribution and use in source and binary forms, with or without
+   modification, are permitted provided that the following conditions are
+   met:
+
+       * Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+       * Redistributions in binary form must reproduce the above
+   copyright notice, this list of conditions and the following disclaimer
+   in the documentation and/or other materials provided with the
+   distribution.
+
+   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+   You can contact the author at :
+   - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* Notice extracted from xxHash homepage :
+
+xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
+It also successfully passes all tests from the SMHasher suite.
+
+Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
+
+Name            Speed       Q.Score   Author
+xxHash          5.4 GB/s     10
+CrapWow         3.2 GB/s      2       Andrew
+MumurHash 3a    2.7 GB/s     10       Austin Appleby
+SpookyHash      2.0 GB/s     10       Bob Jenkins
+SBox            1.4 GB/s      9       Bret Mulvey
+Lookup3         1.2 GB/s      9       Bob Jenkins
+SuperFastHash   1.2 GB/s      1       Paul Hsieh
+CityHash64      1.05 GB/s    10       Pike & Alakuijala
+FNV             0.55 GB/s     5       Fowler, Noll, Vo
+CRC32           0.43 GB/s     9
+MD5-32          0.33 GB/s    10       Ronald L. Rivest
+SHA1-32         0.28 GB/s    10
+
+Q.Score is a measure of quality of the hash function.
+It depends on successfully passing SMHasher test set.
+10 is a perfect score.
+
+A 64-bit version, named XXH64, is available since r35.
+It offers much better speed, but for 64-bit applications only.
+Name     Speed on 64 bits    Speed on 32 bits
+XXH64       13.8 GB/s            1.9 GB/s
+XXH32        6.8 GB/s            6.0 GB/s
+*/
+
+#ifndef XXHASH_H_5627135585666179
+#define XXHASH_H_5627135585666179 1
+
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+
+/* ****************************
+*  Definitions
+******************************/
+#include <stddef.h>   /* size_t */
+typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
+
+
+/* ****************************
+ *  API modifier
+ ******************************/
+/** XXH_INLINE_ALL (and XXH_PRIVATE_API)
+ *  This is useful to include xxhash functions in `static` mode
+ *  in order to inline them, and remove their symbol from the public list.
+ *  Inlining can offer dramatic performance improvement on small keys.
+ *  Methodology :
+ *     #define XXH_INLINE_ALL
+ *     #include "xxhash.h"
+ * `xxhash.c` is automatically included.
+ *  It's not useful to compile and link it as a separate module.
+ */
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  ifndef XXH_STATIC_LINKING_ONLY
+#    define XXH_STATIC_LINKING_ONLY
+#  endif
+#  if defined(__GNUC__)
+#    define XXH_PUBLIC_API static __inline __attribute__((unused))
+#  elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
+#    define XXH_PUBLIC_API static inline
+#  elif defined(_MSC_VER)
+#    define XXH_PUBLIC_API static __inline
+#  else
+     /* this version may generate warnings for unused static functions */
+#    define XXH_PUBLIC_API static
+#  endif
+#else
+#  if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT))
+#    ifdef XXH_EXPORT
+#      define XXH_PUBLIC_API __declspec(dllexport)
+#    elif XXH_IMPORT
+#      define XXH_PUBLIC_API __declspec(dllimport)
+#    endif
+#  else
+#    define XXH_PUBLIC_API   /* do nothing */
+#  endif
+#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */
+
+/*! XXH_NAMESPACE, aka Namespace Emulation :
+ *
+ * If you want to include _and expose_ xxHash functions from within your own library,
+ * but also want to avoid symbol collisions with other libraries which may also include xxHash,
+ *
+ * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
+ * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values).
+ *
+ * Note that no change is required within the calling program as long as it includes `xxhash.h` :
+ * regular symbol name will be automatically translated by this header.
+ */
+#ifdef XXH_NAMESPACE
+#  define XXH_CAT(A,B) A##B
+#  define XXH_NAME2(A,B) XXH_CAT(A,B)
+#  define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
+#  define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
+#  define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
+#  define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
+#  define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
+#  define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
+#  define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
+#  define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
+#  define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
+#  define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
+#  define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
+#  define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
+#  define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
+#  define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
+#  define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
+#  define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
+#  define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
+#  define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
+#  define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
+#endif
+
+
+/* *************************************
+*  Version
+***************************************/
+#define XXH_VERSION_MAJOR    0
+#define XXH_VERSION_MINOR    7
+#define XXH_VERSION_RELEASE  0
+#define XXH_VERSION_NUMBER  (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
+XXH_PUBLIC_API unsigned XXH_versionNumber (void);
+
+
+/*-**********************************************************************
+*  32-bit hash
+************************************************************************/
+typedef unsigned int XXH32_hash_t;
+
+/*! XXH32() :
+    Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input".
+    The memory between input & input+length must be valid (allocated and read-accessible).
+    "seed" can be used to alter the result predictably.
+    Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */
+XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH32_state_s XXH32_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH32_freeState(XXH32_state_t* statePtr);
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset  (XXH32_state_t* statePtr, unsigned int seed);
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH32_hash_t  XXH32_digest (const XXH32_state_t* statePtr);
+
+/*
+ * Streaming functions generate the xxHash of an input provided in multiple segments.
+ * Note that, for small input, they are slower than single-call functions, due to state management.
+ * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized.
+ *
+ * XXH state must first be allocated, using XXH*_createState() .
+ *
+ * Start a new hash by initializing state with a seed, using XXH*_reset().
+ *
+ * Then, feed the hash state by calling XXH*_update() as many times as necessary.
+ * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
+ *
+ * Finally, a hash value can be produced anytime, by using XXH*_digest().
+ * This function returns the nn-bits hash as an int or long long.
+ *
+ * It's still possible to continue inserting input into the hash state after a digest,
+ * and generate some new hashes later on, by calling again XXH*_digest().
+ *
+ * When done, free XXH state space if it was allocated dynamically.
+ */
+
+/*======   Canonical representation   ======*/
+
+typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
+
+/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
+ * The canonical representation uses human-readable write convention, aka big-endian (large digits first).
+ * These functions allow transformation of hash result into and from its canonical format.
+ * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
+ */
+
+
+#ifndef XXH_NO_LONG_LONG
+/*-**********************************************************************
+*  64-bit hash
+************************************************************************/
+typedef unsigned long long XXH64_hash_t;
+
+/*! XXH64() :
+    Calculate the 64-bit hash of sequence of length "len" stored at memory address "input".
+    "seed" can be used to alter the result predictably.
+    This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark).
+*/
+XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH64_state_s XXH64_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH64_freeState(XXH64_state_t* statePtr);
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset  (XXH64_state_t* statePtr, unsigned long long seed);
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH64_hash_t  XXH64_digest (const XXH64_state_t* statePtr);
+
+/*======   Canonical representation   ======*/
+typedef struct { unsigned char digest[8]; } XXH64_canonical_t;
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);
+
+
+#endif  /* XXH_NO_LONG_LONG */
+
+
+
+#ifdef XXH_STATIC_LINKING_ONLY
+
+/* ================================================================================================
+   This section contains declarations which are not guaranteed to remain stable.
+   They may change in future versions, becoming incompatible with a different version of the library.
+   These declarations should only be used with static linking.
+   Never use them in association with dynamic linking !
+=================================================================================================== */
+
+/* These definitions are only present to allow
+ * static allocation of XXH state, on stack or in a struct for example.
+ * Never **ever** use members directly. */
+
+#if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+
+struct XXH32_state_s {
+   uint32_t total_len_32;
+   uint32_t large_len;
+   uint32_t v1;
+   uint32_t v2;
+   uint32_t v3;
+   uint32_t v4;
+   uint32_t mem32[4];
+   uint32_t memsize;
+   uint32_t reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+struct XXH64_state_s {
+   uint64_t total_len;
+   uint64_t v1;
+   uint64_t v2;
+   uint64_t v3;
+   uint64_t v4;
+   uint64_t mem64[4];
+   uint32_t memsize;
+   uint32_t reserved[2];   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+
+# else
+
+struct XXH32_state_s {
+   XXH32_hash_t total_len_32;
+   XXH32_hash_t large_len;
+   XXH32_hash_t v1;
+   XXH32_hash_t v2;
+   XXH32_hash_t v3;
+   XXH32_hash_t v4;
+   XXH32_hash_t mem32[4];
+   XXH32_hash_t memsize;
+   XXH32_hash_t reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+#   ifndef XXH_NO_LONG_LONG  /* remove 64-bit support */
+struct XXH64_state_s {
+   XXH64_hash_t total_len;
+   XXH64_hash_t v1;
+   XXH64_hash_t v2;
+   XXH64_hash_t v3;
+   XXH64_hash_t v4;
+   XXH64_hash_t mem64[4];
+   XXH32_hash_t memsize;
+   XXH32_hash_t reserved[2];     /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+#    endif
+
+# endif
+
+
+/*-**********************************************************************
+*  XXH3
+*  New experimental hash
+************************************************************************/
+#ifndef XXH_NO_LONG_LONG
+
+
+/* ============================================
+ * XXH3 is a new hash algorithm,
+ * featuring vastly improved speed performance
+ * for both small and large inputs.
+ * A full speed analysis will be published,
+ * it requires a lot more space than this comment can handle.
+ * In general, expect XXH3 to run about ~2x faster on large inputs,
+ * and >3x faster on small ones, though exact difference depend on platform.
+ *
+ * The algorithm is portable, will generate the same hash on all platforms.
+ * It benefits greatly from vectorization units, but does not require it.
+ *
+ * XXH3 offers 2 variants, _64bits and _128bits.
+ * When only 64 bits are needed, prefer calling the _64bits variant :
+ * it reduces the amount of mixing, resulting in faster speed on small inputs.
+ * It's also generally simpler to manipulate a scalar type than a struct.
+ * Note : the low 64-bit field of the _128bits variant is the same as _64bits result.
+ *
+ * The XXH3 algorithm is still considered experimental.
+ * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use.
+ * While labelled experimental, the produced result can still change between versions.
+ *
+ * The API currently supports one-shot hashing only.
+ * The full version will include streaming capability, and canonical representation
+ * Long term optional feature may include custom secret keys, and secret key generation.
+ *
+ * There are still a number of opened questions that community can influence during the experimental period.
+ * I'm trying to list a few of them below, though don't consider this list as complete.
+ *
+ * - 128-bits output type : currently defined as a structure of 2 64-bits fields.
+ *                          That's because 128-bit values do not exist in C standard.
+ *                          Note that it means that, at byte level, result is not identical depending on endianess.
+ *                          However, at field level, they are identical on all platforms.
+ *                          The canonical representation will solve the issue of identical byte-level representation across platforms,
+ *                          which is necessary for serialization.
+ *                          Would there be a better representation for a 128-bit hash result ?
+ *                          Are the names of the inner 64-bit fields important ? Should they be changed ?
+ *
+ * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian).
+ *                          What should it be for the 128-bit variant ?
+ *                          Since it's no longer a scalar value, big-endian representation is no longer an obvious choice.
+ *                          One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian)
+ *                          Another one : represent it in the same order as natural order in the struct for little-endian platforms.
+ *                                        Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms.
+ *
+ * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct.
+ *                          Granted, it's not terribly difficult to create a comparator, but it's still a workload.
+ *                          Would it be beneficial to declare and define a comparator function for XXH128_hash_t ?
+ *                          Are there other operations on XXH128_hash_t which would be desirable ?
+ *
+ * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits.
+ *                          This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases.
+ *                          But it might influence the design of XXH128_hash_t, in ways which may block other possibilities.
+ *                          Good idea, bad idea ?
+ *
+ * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant.
+ *                          It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash.
+ *                          Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value.
+ *                          It would either replace current choice, or add a new one.
+ *                          Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`).
+ *                          If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ?
+ *
+ * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed.
+ *                          This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty.
+ *                          Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64).
+ *                          But is it still fine to return the seed when the seed is non-zero ?
+ *                          Are there use case which would depend on this behavior, or would prefer a mixing of the seed ?
+ */
+
+#ifdef XXH_NAMESPACE
+#  define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128)
+#  define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits)
+#  define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed)
+#  define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits)
+#  define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed)
+#endif
+
+
+typedef struct {
+    XXH64_hash_t low64;
+    XXH64_hash_t high64;
+} XXH128_hash_t;
+
+XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed);
+
+/* note : variants without seed produce same result as variant with seed == 0 */
+XXH_PUBLIC_API XXH64_hash_t  XXH3_64bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH64_hash_t  XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed);  /* == XXH128() */
+
+
+#endif  /* XXH_NO_LONG_LONG */
+
+
+/*-**********************************************************************
+*  XXH_INLINE_ALL
+************************************************************************/
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  include "xxhash.c"   /* include xxhash function bodies as `static`, for inlining */
+#endif
+
+
+
+#endif /* XXH_STATIC_LINKING_ONLY */
+
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* XXHASH_H_5627135585666179 */
-- 
2.16.4

[RFC PATCH 17/17] btrfs-progs: add xxhash64 as checksum algorithm

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

Add xxhash64 as another checksumming algorithm.

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 Makefile                  |  3 ++-
 cmds/inspect-dump-super.c | 25 ++++++++++++++++---------
 convert/common.c          |  2 +-
 convert/main.c            |  2 +-
 crypto/hash.c             | 16 ++++++++++++++++
 crypto/hash.h             | 10 ++++++++++
 crypto/xxhash.c           |  2 +-
 ctree.h                   | 18 +++++++++++++-----
 disk-io.c                 |  7 +++++--
 image/main.c              |  5 +++--
 mkfs/common.c             | 14 +++++++-------
 mkfs/main.c               |  6 +++++-
 12 files changed, 80 insertions(+), 30 deletions(-)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h

diff --git a/Makefile b/Makefile
index 82417d19a9f8..1982e6f5d70e 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
 	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
 		   kernel-lib/crc32c.o common/messages.o \
-		   uuid-tree.o utils-lib.o common/rbtree-utils.o
+		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
+		   crypto/hash.o crypto/xxhash.o
 libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
 	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
 	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 58bf82b0bbd3..1001c8aa5c85 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
 				     super_flags_num, BTRFS_SUPER_FLAG_SUPP);
 }
 
+static bool is_valid_csum_type(u16 csum_type)
+{
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void dump_superblock(struct btrfs_super_block *sb, int full)
 {
 	int i;
@@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	csum_type = btrfs_super_csum_type(sb);
 	csum_size = BTRFS_CSUM_SIZE;
 	printf("csum_type\t\t%hu (", csum_type);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
 		printf("INVALID");
 	} else {
-		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
-			printf("crc32c");
-			csum_size = btrfs_csum_sizes[csum_type];
-		} else {
-			printf("unknown");
-		}
+		printf("%s", btrfs_csums[csum_type].name);
+		csum_size = btrfs_csums[csum_type].size;
 	}
 	printf(")\n");
 	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
@@ -342,8 +349,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	printf("csum\t\t\t0x");
 	for (i = 0, p = sb->csum; i < csum_size; i++)
 		printf("%02x", p[i]);
-	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
-	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
+	if (!is_valid_csum_type(csum_type) ||
+	    csum_size != btrfs_csums[csum_type].size)
 		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
 	else if (check_csum_sblock(sb, csum_size, csum_type))
 		printf(" [match]");
diff --git a/convert/common.c b/convert/common.c
index 8cae507ec0f7..2b9613f01787 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
 {
 	int ret;
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 
 	/* Temporary extent buffer is always mapped 1:1 on disk */
diff --git a/convert/main.c b/convert/main.c
index 5e6b12431f59..5eb2a59fb68a 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
 	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
 	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+	csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
 			     btrfs_super_csum_type(super));
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 		BTRFS_SUPER_INFO_OFFSET);
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..fda7fc4e9f23
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,16 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+	XXH64_hash_t hash;
+
+	hash = XXH64(buf, length, 0);
+	/* NOTE: we're not taking the canonical form here but the plain hash to
+	 * be compatible with the kernel implementation!
+	 */
+	memcpy(out, &hash, 8);
+
+	return 0;
+}
+
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX	32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/crypto/xxhash.c b/crypto/xxhash.c
index af9d02795ac6..7f381c8b56a0 100644
--- a/crypto/xxhash.c
+++ b/crypto/xxhash.c
@@ -1018,7 +1018,7 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src
 *  New generation hash designed for speed on small keys and vectorization
 ************************************************************************ */
 
-#include "xxh3.h"
+/* #include "xxh3.h" */
 
 
 #endif  /* XXH_NO_LONG_LONG */
diff --git a/ctree.h b/ctree.h
index c63fad8310db..bb10080c9238 100644
--- a/ctree.h
+++ b/ctree.h
@@ -165,10 +165,18 @@ struct btrfs_free_space_ctl;
 #define BTRFS_CSUM_SIZE 32
 
 /* csum types */
-#define BTRFS_CSUM_TYPE_CRC32	0
+enum btrfs_csum_type {
+	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
+};
 
-/* four bytes for CRC32 */
-static int btrfs_csum_sizes[] = { 4 };
+static struct btrfs_csum {
+	u16 size;
+	const char *name;
+} btrfs_csums[] = {
+	[BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+	[BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
+};
 
 #define BTRFS_EMPTY_DIR_SIZE 0
 
@@ -2264,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
 static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
 {
 	int t = btrfs_super_csum_type(s);
-	BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
-	return btrfs_csum_sizes[t];
+	BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
+	return btrfs_csums[t].size;
 }
 
 static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
diff --git a/disk-io.c b/disk-io.c
index 810c2e14294a..ce0b746f4db9 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
 #include "print-tree.h"
 #include "common/rbtree-utils.h"
 #include "common/device-scan.h"
+#include "crypto/hash.h"
 
 /* specified errno for check_tree_block */
 #define BTRFS_BAD_BYTENR		(-1)
@@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 		crc = crc32c(crc, data, len);
 		put_unaligned_le32(~crc, out);
 		return 0;
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return hash_xxhash(data, len, out);
 	default:
 		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
 		ASSERT(0);
@@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	}
 
 	csum_type = btrfs_super_csum_type(sb);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
 		error("unsupported checksum algorithm %u", csum_type);
 		return -EIO;
 	}
-	csum_size = btrfs_csum_sizes[csum_type];
+	csum_size = btrfs_csums[csum_type].size;
 
 	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
 			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
diff --git a/image/main.c b/image/main.c
index 0c8ffede56f5..1265152cf524 100644
--- a/image/main.c
+++ b/image/main.c
@@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
 
 static void csum_block(u8 *buf, size_t len)
 {
-	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
+	u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
+	u8 result[csum_size];
 	u32 crc = ~(u32)0;
 	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
 	put_unaligned_le32(~crc, result);
-	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
+	memcpy(buf, result, csum_size);
 }
 
 static int has_name(struct btrfs_key *key)
diff --git a/mkfs/common.c b/mkfs/common.c
index 4a417bd7a306..939be5eb2dc2 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 
 	/* write back root tree */
@@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
@@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
@@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
@@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
@@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
@@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
diff --git a/mkfs/main.c b/mkfs/main.c
index 3deff76045ba..4d244d56bbac 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -385,6 +385,9 @@ static u16 parse_csum_type(const char *s)
 {
 	if (strcasecmp(s, "crc32c") == 0) {
 		return BTRFS_CSUM_TYPE_CRC32;
+	} else if (strcasecmp(s, "xxhash64") == 0 ||
+		   strcasecmp(s, "xxhash") == 0) {
+		return BTRFS_CSUM_TYPE_XXHASH;
 	} else {
 		error("unknown csum type %s", s);
 		exit(1);
@@ -1370,7 +1373,8 @@ raid_groups:
 			pretty_size(allocation.system));
 		printf("SSD detected:       %s\n", ssd ? "yes" : "no");
 		btrfs_parse_features_to_string(features_buf, features);
-		printf("Incompat features:  %s", features_buf);
+		printf("Incompat features:  %s\n", features_buf);
+		printf("Checksum:           %s", btrfs_csums[csum_type].name);
 		printf("\n");
 
 		list_all_devices(root);
-- 
2.16.4