* [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 12:56 ` David Sterba
2019-09-24 13:06 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Johannes Thumshirn
` (11 subsequent siblings)
12 siblings, 2 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
The callers of csum_tree_block_size() blindly assume we're only having
crc32c as a possible checksum and thus pass in
btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
csum_tree_block_size().
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
mkfs/common.c | 14 +++++++-------
mkfs/common.h | 2 ++
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/mkfs/common.c b/mkfs/common.c
index caca5e707233..b6e549b19272 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
/* write back root tree */
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
if (ret != BTRFS_SUPER_INFO_SIZE) {
diff --git a/mkfs/common.h b/mkfs/common.h
index 28912906d0a9..1ca71a4fcce5 100644
--- a/mkfs/common.h
+++ b/mkfs/common.h
@@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
u64 features;
/* Size of the filesystem in bytes */
u64 num_bytes;
+ /* checksum algorithm to use */
+ enum btrfs_csum_type csum_type;
/* Output fields, set during creation */
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
@ 2019-09-24 12:56 ` David Sterba
2019-09-24 13:06 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 12:56 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> index 28912906d0a9..1ca71a4fcce5 100644
> --- a/mkfs/common.h
> +++ b/mkfs/common.h
> @@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
> u64 features;
> /* Size of the filesystem in bytes */
> u64 num_bytes;
> + /* checksum algorithm to use */
> + enum btrfs_csum_type csum_type;
This is defined in the following patch so the compilation breaks here,
I'll see if reordering 1 and 2 fixes that.
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
2019-09-24 12:56 ` David Sterba
@ 2019-09-24 13:06 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 13:06 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> mkfs/common.c | 14 +++++++-------
> mkfs/common.h | 2 ++
> 2 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/mkfs/common.c b/mkfs/common.c
> index caca5e707233..b6e549b19272 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
I don't see where cfg->csum_type is initialized. The tests pass so
there's probably some implicit initialization to 0 that makes it work.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Johannes Thumshirn
` (10 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Cache the super-block's checksum type field in 'struct recover_control'.
This will be needed for further refactoring the checksum handling.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
cmds/rescue-chunk-recover.c | 2 ++
ctree.h | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index c9b268781159..1959a2047c17 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -47,6 +47,7 @@ struct recover_control {
int yes;
u16 csum_size;
+ u16 csum_type;
u32 sectorsize;
u32 nodesize;
u64 generation;
@@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path)
rc->generation = btrfs_super_generation(sb);
rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb);
rc->csum_size = btrfs_super_csum_size(sb);
+ rc->csum_type = btrfs_super_csum_type(sb);
/* if seed, the result of scanning below will be partial */
if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) {
diff --git a/ctree.h b/ctree.h
index 0d12563b7261..870d9f4948de 100644
--- a/ctree.h
+++ b/ctree.h
@@ -165,7 +165,9 @@ struct btrfs_free_space_ctl;
#define BTRFS_CSUM_SIZE 32
/* csum types */
-#define BTRFS_CSUM_TYPE_CRC32 0
+enum btrfs_csum_type {
+ BTRFS_CSUM_TYPE_CRC32 = 0,
+};
/* four bytes for CRC32 */
static int btrfs_csum_sizes[] = { 4 };
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Johannes Thumshirn
` (9 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
and verify_tree_block_csum_silent().
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-corrupt-block.c | 3 ++-
cmds/rescue-chunk-recover.c | 3 ++-
convert/common.c | 3 ++-
convert/main.c | 3 ++-
disk-io.c | 21 ++++++++++++---------
disk-io.h | 5 +++--
mkfs/common.c | 21 ++++++++++++++-------
7 files changed, 37 insertions(+), 22 deletions(-)
diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index bbef0c02e5d1..1dde9594bdcc 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
if (!trans) {
u16 csum_size =
btrfs_super_csum_size(fs_info->super_copy);
- csum_tree_block_size(eb, csum_size, 0);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+ csum_tree_block_size(eb, csum_size, 0, csum_type);
write_extent_to_disk(eb);
}
}
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 1959a2047c17..c2a26ad040ee 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
continue;
}
- if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
+ if (verify_tree_block_csum_silent(buf, rc->csum_size,
+ rc->csum_type)) {
bytenr += rc->sectorsize;
continue;
}
diff --git a/convert/common.c b/convert/common.c
index dea5f5b20d50..f8bbb23cba89 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
{
int ret;
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
/* Temporary extent buffer is always mapped 1:1 on disk */
ret = pwrite(fd, buf->data, buf->len, bytenr);
diff --git a/convert/main.c b/convert/main.c
index 9711874bd137..5e6b12431f59 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+ btrfs_super_csum_type(super));
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
BTRFS_SUPER_INFO_OFFSET);
if (ret != BTRFS_SUPER_INFO_SIZE)
diff --git a/disk-io.c b/disk-io.c
index 01314504a50a..a4995a628210 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
- int verify, int silent)
+ int verify, int silent, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
u32 len;
@@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
return 0;
}
-int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
+int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
+ u16 csum_type)
{
- return __csum_tree_block_size(buf, csum_size, verify, 0);
+ return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
}
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+ u16 csum_type)
{
- return __csum_tree_block_size(buf, csum_size, 1, 1);
+ return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
}
int csum_tree_block(struct btrfs_fs_info *fs_info,
struct extent_buffer *buf, int verify)
{
- u16 csum_size =
- btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
if (verify && fs_info->suppress_check_block_errors)
- return verify_tree_block_csum_silent(buf, csum_size);
- return csum_tree_block_size(buf, csum_size, verify);
+ return verify_tree_block_csum_silent(buf, csum_size, csum_type);
+ return csum_tree_block_size(buf, csum_size, verify, csum_type);
}
struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
diff --git a/disk-io.h b/disk-io.h
index 7b5c3806ba98..394997ad72cb 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
- int verify);
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
+ int verify, u16 csum_type);
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+ u16 csum_type);
int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
int write_tree_block(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info,
diff --git a/mkfs/common.c b/mkfs/common.c
index b6e549b19272..9762391a8d2b 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
/* write back root tree */
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
if (ret != BTRFS_SUPER_INFO_SIZE) {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (2 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Johannes Thumshirn
` (8 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-sb-mod.c | 9 ++++++---
check/main.c | 2 +-
cmds/inspect-dump-super.c | 3 ++-
cmds/rescue-chunk-recover.c | 2 +-
convert/common.c | 3 ++-
disk-io.c | 12 ++++++------
disk-io.h | 2 +-
file-item.c | 2 +-
8 files changed, 20 insertions(+), 15 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 16a26f772494..932c2a0432ef 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb)
u32 crc = ~(u32)0;
crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
- crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
return !memcmp(sb, &result, csum_size);
@@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb)
u32 crc = ~(u32)0;
if (is_sb) {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
} else {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BLOCKSIZE - BTRFS_CSUM_SIZE);
}
btrfs_csum_final(crc, result);
diff --git a/check/main.c b/check/main.c
index 2e16b4e6f05b..0f02c85368ec 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5660,7 +5660,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
tmp = offset + data_checked;
csum = btrfs_csum_data((char *)data + tmp,
- csum, fs_info->sectorsize);
+ (u8 *)&csum, fs_info->sectorsize);
btrfs_csum_final(csum, (u8 *)&csum);
csum_offset = leaf_offset +
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 65fb3506eac6..96ad3deca3d8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size)
u32 crc = ~(u32)0;
crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
- crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
return !memcmp(sb, &result, csum_size);
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index c2a26ad040ee..37c024ed8965 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(data, csum_result, len);
+ csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
btrfs_csum_final(csum_result, (u8 *)&csum_result);
if (csum_result != tree_csum)
ret = 1;
diff --git a/convert/common.c b/convert/common.c
index f8bbb23cba89..ab8e6b9f4749 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u32 crc = ~(u32)0;
int ret;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
diff --git a/disk-io.c b/disk-io.c
index a4995a628210..a0c37c569d58 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(char *data, u32 seed, size_t len)
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
{
- return crc32c(seed, data, len);
+ return crc32c(*(u32*)seed, data, len);
}
void btrfs_csum_final(u32 crc, u8 *result)
@@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len);
+ crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
btrfs_csum_final(crc, result);
if (verify) {
@@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
csum_size = btrfs_csum_sizes[csum_type];
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
@@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
@@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
diff --git a/disk-io.h b/disk-io.h
index 394997ad72cb..92c87f28f8b2 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u32 seed, size_t len);
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
void btrfs_csum_final(u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
diff --git a/file-item.c b/file-item.c
index 3bf48c68913d..5f6548e9a74f 100644
--- a/file-item.c
+++ b/file-item.c
@@ -312,7 +312,7 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(data, csum_result, len);
+ csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
btrfs_csum_final(csum_result, (u8 *)&csum_result);
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final()
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (3 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Johannes Thumshirn
` (7 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
In preparation to supporting new checksum algorithm pass the checksum type
to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate any
differences in processing into the respective functions
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Changes to v1:
- Update changelog and squashed the other identical patch
---
btrfs-sb-mod.c | 15 ++++++++++-----
check/main.c | 6 ++++--
cmds/inspect-dump-super.c | 8 ++++----
cmds/rescue-chunk-recover.c | 10 ++++++----
convert/common.c | 5 +++--
disk-io.c | 39 +++++++++++++++++++++++++++------------
disk-io.h | 4 ++--
file-item.c | 7 +++++--
free-space-cache.c | 2 +-
image/main.c | 2 +-
10 files changed, 63 insertions(+), 35 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 932c2a0432ef..d9630f187d0f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb)
{
u8 result[csum_size];
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(sb);
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
return !memcmp(sb, &result, csum_size);
}
@@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb)
u8 result[csum_size];
struct btrfs_header *hdr;
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(block);
if (is_sb) {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)block + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
} else {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)block + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BLOCKSIZE - BTRFS_CSUM_SIZE);
}
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index 0f02c85368ec..1896a28f157d 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5619,6 +5619,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
struct btrfs_fs_info *fs_info = root->fs_info;
u64 offset = 0;
u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
char *data;
unsigned long csum_offset;
u32 csum;
@@ -5659,9 +5660,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
csum = ~(u32)0;
tmp = offset + data_checked;
- csum = btrfs_csum_data((char *)data + tmp,
+ csum = btrfs_csum_data(csum_type,
+ (char *)data + tmp,
(u8 *)&csum, fs_info->sectorsize);
- btrfs_csum_final(csum, (u8 *)&csum);
+ btrfs_csum_final(csum_type, csum, (u8 *)&csum);
csum_offset = leaf_offset +
tmp / fs_info->sectorsize * csum_size;
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 96ad3deca3d8..40019a6670ef 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -35,15 +35,15 @@
#include "kernel-lib/crc32c.h"
#include "common/help.h"
-static int check_csum_sblock(void *sb, int csum_size)
+static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
u32 crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
return !memcmp(sb, &result, csum_size);
}
@@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
printf(" [UNKNOWN CSUM TYPE OR SIZE]");
- else if (check_csum_sblock(sb, csum_size))
+ else if (check_csum_sblock(sb, csum_size, csum_type))
printf(" [match]");
else
printf(" [DON'T MATCH]");
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 37c024ed8965..d556323ec34a 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
return dev_offset + data_offset;
}
-static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
+static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
+ u16 csum_type)
{
char *data;
int ret = 0;
@@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_result, (u8 *)&csum_result);
+ csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+ btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
if (csum_result != tree_csum)
ret = 1;
out:
@@ -2102,7 +2103,8 @@ next_csum:
devext->objectid, 1));
ret = check_one_csum(dev->fd, data_offset, blocksize,
- tree_csum);
+ tree_csum,
+ btrfs_super_csum_type(root->fs_info->super_copy));
if (ret < 0)
goto fail_out;
else if (ret > 0)
diff --git a/convert/common.c b/convert/common.c
index ab8e6b9f4749..894a6ee0ba90 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u64 sb_bytenr)
{
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(sb);
int ret;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
if (ret < BTRFS_SUPER_INFO_SIZE)
ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index a0c37c569d58..7e538969c57a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
{
- return crc32c(*(u32*)seed, data, len);
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ return crc32c(*(u32*)seed, data, len);
+ default: /* Not reached */
+ return ~(u32)0;
+ }
+
}
-void btrfs_csum_final(u32 crc, u8 *result)
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
{
- put_unaligned_le32(~crc, result);
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ put_unaligned_le32(~crc, result);
+ break;
+ default: /* Not reached */
+ break;
+ }
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
- btrfs_csum_final(crc, result);
+ crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
+ (u8 *)&crc, len);
+ btrfs_csum_final(csum_type, crc, result);
if (verify) {
if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
@@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
csum_size = btrfs_csum_sizes[csum_type];
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
if (memcmp(result, sb->csum, csum_size)) {
error("superblock checksum mismatch");
@@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
u64 bytenr;
u32 crc;
int i, ret;
+ u16 csum_type = btrfs_super_csum_type(sb);
/*
* We need to write super block after all metadata written.
@@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 92c87f28f8b2..4b5e9ea86385 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u32 crc, u8 *result);
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 5f6548e9a74f..78fdcecd0bab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
u16 csum_size =
btrfs_super_csum_size(root->fs_info->super_copy);
+ u16 csum_type =
+ btrfs_super_csum_type(root->fs_info->super_copy);
+
path = btrfs_alloc_path();
if (!path)
return -ENOMEM;
@@ -312,8 +315,8 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_result, (u8 *)&csum_result);
+ csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+ btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
(unsigned long long)bytenr);
diff --git a/free-space-cache.c b/free-space-cache.c
index e872eb6a00db..8a57f86dc650 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
io_ctl_map_page(io_ctl, 0);
crc = crc32c(crc, io_ctl->orig + offset,
io_ctl->root->fs_info->sectorsize - offset);
- btrfs_csum_final(crc, (u8 *)&crc);
+ put_unaligned_le32(~crc, (u8 *)&crc);
if (val != crc) {
printk("btrfs: csum mismatch on free space cache\n");
io_ctl_unmap_page(io_ctl);
diff --git a/image/main.c b/image/main.c
index 28ef1609b5ff..0c8ffede56f5 100644
--- a/image/main.c
+++ b/image/main.c
@@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
u32 crc = ~(u32)0;
crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ put_unaligned_le32(~crc, result);
memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
}
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (4 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 07/12] btrfs-progs: update checksumming api Johannes Thumshirn
` (6 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb'
argument set to 1.
Get rid of the special case for is_sb == 0.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
btrfs-sb-mod.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index d9630f187d0f..9e64b34d2b8f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb)
return !memcmp(sb, &result, csum_size);
}
-static void update_block_csum(void *block, int is_sb)
+static void update_block_csum(void *block)
{
u8 result[csum_size];
struct btrfs_header *hdr;
u32 crc = ~(u32)0;
u16 csum_type = btrfs_super_csum_type(block);
- if (is_sb) {
- crc = btrfs_csum_data(csum_type,
- (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- } else {
- crc = btrfs_csum_data(csum_type,
- (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BLOCKSIZE - BTRFS_CSUM_SIZE);
- }
+ crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+
btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
@@ -354,7 +347,7 @@ int main(int argc, char **argv) {
if (changed) {
printf("Update csum\n");
- update_block_csum(buf, 1);
+ update_block_csum(buf);
ret = pwrite(fd, buf, BLOCKSIZE, off);
if (ret <= 0) {
printf("pwrite error %d at offset %llu\n",
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 07/12] btrfs-progs: update checksumming api
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (5 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
` (5 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
Update the checksumming API to be able to cope with more checksum types
than just CRC32C.
Co-developed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-sb-mod.c | 21 ++++++----------
check/main.c | 20 +++++++--------
cmds/inspect-dump-super.c | 9 +++----
cmds/rescue-chunk-recover.c | 12 ++++++---
convert/common.c | 9 +++----
disk-io.c | 59 ++++++++++++++++++---------------------------
disk-io.h | 3 +--
file-item.c | 8 +++---
8 files changed, 60 insertions(+), 81 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 9e64b34d2b8f..105b556b0cf1 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -33,31 +33,24 @@ static int csum_size;
static int check_csum_superblock(void *sb)
{
- u8 result[csum_size];
- u32 crc = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
u16 csum_type = btrfs_super_csum_type(sb);
- crc = btrfs_csum_data(csum_type,
- (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- return !memcmp(sb, &result, csum_size);
+ return !memcmp(sb, result, csum_size);
}
static void update_block_csum(void *block)
{
- u8 result[csum_size];
+ u8 result[BTRFS_CSUM_SIZE];
struct btrfs_header *hdr;
- u32 crc = ~(u32)0;
u16 csum_type = btrfs_super_csum_type(block);
- crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index 1896a28f157d..5028599a773a 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5620,10 +5620,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
u64 offset = 0;
u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
- char *data;
+ u8 *data;
unsigned long csum_offset;
- u32 csum;
- u32 csum_expected;
+ u8 result[BTRFS_CSUM_SIZE];
+ u8 csum_expected[BTRFS_CSUM_SIZE];
u64 read_len;
u64 data_checked = 0;
u64 tmp;
@@ -5649,7 +5649,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
for (mirror = 1; mirror <= num_copies; mirror++) {
read_len = num_bytes - offset;
/* read as much space once a time */
- ret = read_extent_data(fs_info, data + offset,
+ ret = read_extent_data(fs_info, (char *)data + offset,
bytenr + offset, &read_len, mirror);
if (ret)
goto out;
@@ -5657,24 +5657,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
data_checked = 0;
/* verify every 4k data's checksum */
while (data_checked < read_len) {
- csum = ~(u32)0;
tmp = offset + data_checked;
- csum = btrfs_csum_data(csum_type,
- (char *)data + tmp,
- (u8 *)&csum, fs_info->sectorsize);
- btrfs_csum_final(csum_type, csum, (u8 *)&csum);
+ btrfs_csum_data(csum_type, data + tmp,
+ result, fs_info->sectorsize);
csum_offset = leaf_offset +
tmp / fs_info->sectorsize * csum_size;
read_extent_buffer(eb, (char *)&csum_expected,
csum_offset, csum_size);
- if (csum != csum_expected) {
+ if (memcmp(result, csum_expected, csum_size) != 0) {
csum_mismatch = true;
+ /* FIXME: format */
fprintf(stderr,
"mirror %d bytenr %llu csum %u expected csum %u\n",
mirror, bytenr + tmp,
- csum, csum_expected);
+ result[0], csum_expected[0]);
}
data_checked += fs_info->sectorsize;
}
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 40019a6670ef..58bf82b0bbd3 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -38,14 +38,11 @@
static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
- u32 crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- return !memcmp(sb, &result, csum_size);
+ return !memcmp(sb, result, csum_size);
}
static void print_sys_chunk_array(struct btrfs_super_block *sb)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index d556323ec34a..329a608dfc6b 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
{
char *data;
int ret = 0;
- u32 csum_result = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
+ int csum_size = 0;
+ u8 expected_csum[BTRFS_CSUM_SIZE];
+
+ ASSERT(0);
data = malloc(len);
if (!data)
@@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
- if (csum_result != tree_csum)
+ put_unaligned_le32(tree_csum, expected_csum);
+ btrfs_csum_data(csum_type, (u8 *)data, result, len);
+ if (memcmp(result, expected_csum, csum_size) != 0)
ret = 1;
out:
free(data);
diff --git a/convert/common.c b/convert/common.c
index 894a6ee0ba90..8cae507ec0f7 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u64 sb_bytenr)
{
- u32 crc = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
u16 csum_type = btrfs_super_csum_type(sb);
int ret;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
if (ret < BTRFS_SUPER_INFO_SIZE)
ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index 7e538969c57a..810c2e14294a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
{
- switch (csum_type) {
- case BTRFS_CSUM_TYPE_CRC32:
- return crc32c(*(u32*)seed, data, len);
- default: /* Not reached */
- return ~(u32)0;
- }
-
-}
+ u32 crc = ~(u32)0;
+ memset(out, 0, BTRFS_CSUM_SIZE);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
-{
switch (csum_type) {
case BTRFS_CSUM_TYPE_CRC32:
- put_unaligned_le32(~crc, result);
- break;
- default: /* Not reached */
- break;
+ crc = crc32c(crc, data, len);
+ put_unaligned_le32(~crc, out);
+ return 0;
+ default:
+ fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
+ ASSERT(0);
}
+
+ return -1;
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
{
u8 result[BTRFS_CSUM_SIZE];
u32 len;
- u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
- (u8 *)&crc, len);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE,
+ result, len);
if (verify) {
if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
+ /* FIXME: format */
if (!silent)
printk("checksum verify failed on %llu found %08X wanted %08X\n",
(unsigned long long)buf->start,
- *((u32 *)result),
- *((u32*)(char *)buf->data));
+ result[0],
+ buf->data[0]);
return 1;
}
} else {
@@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
{
u8 result[BTRFS_CSUM_SIZE];
- u32 crc;
u16 csum_type;
int csum_size;
u8 *metadata_uuid;
@@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
}
csum_size = btrfs_csum_sizes[csum_type];
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
if (memcmp(result, sb->csum, csum_size)) {
error("superblock checksum mismatch");
@@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
struct btrfs_device *device)
{
u64 bytenr;
- u32 crc;
+ u8 result[BTRFS_CSUM_SIZE];
int i, ret;
u16 csum_type = btrfs_super_csum_type(sb);
@@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
}
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 4b5e9ea86385..545cacda9a79 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 78fdcecd0bab..c6e9d212bcab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
struct btrfs_csum_item *item;
struct extent_buffer *leaf = NULL;
u64 csum_offset;
- u32 csum_result = ~(u32)0;
+ u8 csum_result[BTRFS_CSUM_SIZE];
u32 sectorsize = root->fs_info->sectorsize;
u32 nritems;
u32 ins_size;
@@ -315,14 +315,14 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
+ btrfs_csum_data(csum_type, (u8 *)data, csum_result, len);
+ /* FIXME: does not make sense for non-crc32c */
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
(unsigned long long)bytenr);
}
- write_extent_buffer(leaf, &csum_result, (unsigned long)item,
+ write_extent_buffer(leaf, csum_result, (unsigned long)item,
csum_size);
btrfs_mark_buffer_dirty(path->nodes[0]);
fail:
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (6 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 07/12] btrfs-progs: update checksumming api Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 09/12] btrfs-progs: add xxhash sources Johannes Thumshirn
` (4 subsequent siblings)
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add an option to mkfs to specify which checksum algorithm will be used for
the filesystem.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
convert/common.c | 2 +-
mkfs/common.c | 2 +-
mkfs/main.c | 21 ++++++++++++++++++++-
3 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/convert/common.c b/convert/common.c
index 8cae507ec0f7..2e2318a5863e 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -135,7 +135,7 @@ static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
super->__unused_leafsize = cpu_to_le32(cfg->nodesize);
btrfs_set_super_nodesize(super, cfg->nodesize);
btrfs_set_super_stripesize(super, cfg->stripesize);
- btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
+ btrfs_set_super_csum_type(super, cfg->csum_type);
btrfs_set_super_chunk_root(super, chunk_bytenr);
btrfs_set_super_cache_generation(super, -1);
btrfs_set_super_incompat_flags(super, cfg->features);
diff --git a/mkfs/common.c b/mkfs/common.c
index 9762391a8d2b..4a417bd7a306 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -202,7 +202,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
btrfs_set_super_nodesize(&super, cfg->nodesize);
btrfs_set_super_stripesize(&super, cfg->stripesize);
- btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32);
+ btrfs_set_super_csum_type(&super, cfg->csum_type);
btrfs_set_super_chunk_root_generation(&super, 1);
btrfs_set_super_cache_generation(&super, -1);
btrfs_set_super_incompat_flags(&super, cfg->features);
diff --git a/mkfs/main.c b/mkfs/main.c
index b752da13aba9..e96cbc5399a2 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -352,6 +352,7 @@ static void print_usage(int ret)
printf("\t--shrink (with --rootdir) shrink the filled filesystem to minimal size\n");
printf("\t-K|--nodiscard do not perform whole device TRIM\n");
printf("\t-f|--force force overwrite of existing filesystem\n");
+ printf("\t-C|--checksum checksum algorithm to use (default: crc32c)\n");
printf(" general:\n");
printf("\t-q|--quiet no messages except errors\n");
printf("\t-V|--version print the mkfs.btrfs version and exit\n");
@@ -386,6 +387,18 @@ static u64 parse_profile(const char *s)
return 0;
}
+static enum btrfs_csum_type parse_csum_type(const char *s)
+{
+ if (strcasecmp(s, "crc32c") == 0) {
+ return BTRFS_CSUM_TYPE_CRC32;
+ } else {
+ error("unknown csum type %s", s);
+ exit(1);
+ }
+ /* not reached */
+ return 0;
+}
+
static char *parse_label(const char *input)
{
int len = strlen(input);
@@ -832,6 +845,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
struct mkfs_allocation allocation = { 0 };
struct btrfs_mkfs_config mkfs_cfg;
+ enum btrfs_csum_type csum_type = BTRFS_CSUM_TYPE_CRC32;
crc32c_optimization_init();
@@ -841,6 +855,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
static const struct option long_options[] = {
{ "alloc-start", required_argument, NULL, 'A'},
{ "byte-count", required_argument, NULL, 'b' },
+ { "checksum", required_argument, NULL, 'C' },
{ "force", no_argument, NULL, 'f' },
{ "leafsize", required_argument, NULL, 'l' },
{ "label", required_argument, NULL, 'L'},
@@ -860,7 +875,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
{ NULL, 0, NULL, 0}
};
- c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+ c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq",
long_options, NULL);
if (c < 0)
break;
@@ -938,6 +953,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
case GETOPT_VAL_SHRINK:
shrink_rootdir = true;
break;
+ case 'C':
+ csum_type = parse_csum_type(optarg);
+ break;
case GETOPT_VAL_HELP:
default:
print_usage(c != GETOPT_VAL_HELP);
@@ -1176,6 +1194,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
mkfs_cfg.sectorsize = sectorsize;
mkfs_cfg.stripesize = stripesize;
mkfs_cfg.features = features;
+ mkfs_cfg.csum_type = csum_type;
ret = make_btrfs(fd, &mkfs_cfg);
if (ret) {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
@ 2019-09-24 14:26 ` David Sterba
2019-09-24 15:34 ` Adam Borowski
0 siblings, 1 reply; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:26 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
I'll change the option to '-c' so we have the most common options as
lowercase letters.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-24 14:26 ` David Sterba
@ 2019-09-24 15:34 ` Adam Borowski
2019-09-24 15:52 ` David Sterba
0 siblings, 1 reply; 23+ messages in thread
From: Adam Borowski @ 2019-09-24 15:34 UTC (permalink / raw)
To: dsterba, Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote:
> On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> > Add an option to mkfs to specify which checksum algorithm will be used for
> > the filesystem.
> >
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>
> I'll change the option to '-c' so we have the most common options as
> lowercase letters.
-c is used for compression elsewhere, I'd rather avoid this confusion.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-24 15:34 ` Adam Borowski
@ 2019-09-24 15:52 ` David Sterba
0 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 15:52 UTC (permalink / raw)
To: Adam Borowski
Cc: dsterba, Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 24, 2019 at 05:34:11PM +0200, Adam Borowski wrote:
> On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote:
> > On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> > > Add an option to mkfs to specify which checksum algorithm will be used for
> > > the filesystem.
> > >
> > > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> >
> > I'll change the option to '-c' so we have the most common options as
> > lowercase letters.
>
> -c is used for compression elsewhere, I'd rather avoid this confusion.
Fair point.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 09/12] btrfs-progs: add xxhash sources
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (7 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
` (3 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
git://github.com/Cyan4973/xxHash
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
crypto/xxhash.c | 1025 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
crypto/xxhash.h | 445 ++++++++++++++++++++++++
2 files changed, 1470 insertions(+)
create mode 100644 crypto/xxhash.c
create mode 100644 crypto/xxhash.h
diff --git a/crypto/xxhash.c b/crypto/xxhash.c
new file mode 100644
index 000000000000..58bb749d3078
--- /dev/null
+++ b/crypto/xxhash.c
@@ -0,0 +1,1025 @@
+/*
+* xxHash - Fast Hash algorithm
+* Copyright (C) 2012-2016, Yann Collet
+*
+* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+*
+* Redistribution and use in source and binary forms, with or without
+* modification, are permitted provided that the following conditions are
+* met:
+*
+* * Redistributions of source code must retain the above copyright
+* notice, this list of conditions and the following disclaimer.
+* * Redistributions in binary form must reproduce the above
+* copyright notice, this list of conditions and the following disclaimer
+* in the documentation and/or other materials provided with the
+* distribution.
+*
+* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*
+* You can contact the author at :
+* - xxHash homepage: http://www.xxhash.com
+* - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+
+/* *************************************
+* Tuning parameters
+***************************************/
+/*!XXH_FORCE_MEMORY_ACCESS :
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
+ * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
+ * Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
+ * It can generate buggy code on targets which do not support unaligned memory accesses.
+ * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See http://stackoverflow.com/a/32095106/646947 for details.
+ * Prefer these methods in priority order (0 > 1 > 2)
+ */
+#ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */
+# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
+ || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
+ || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+# define XXH_FORCE_MEMORY_ACCESS 2
+# elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \
+ (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
+ || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
+ || defined(__ARM_ARCH_7S__) ))
+# define XXH_FORCE_MEMORY_ACCESS 1
+# endif
+#endif
+
+/*!XXH_ACCEPT_NULL_INPUT_POINTER :
+ * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault.
+ * When this macro is enabled, xxHash actively checks input for null pointer.
+ * It it is, result for null input pointers is the same as a null-length input.
+ */
+#ifndef XXH_ACCEPT_NULL_INPUT_POINTER /* can be defined externally */
+# define XXH_ACCEPT_NULL_INPUT_POINTER 0
+#endif
+
+/*!XXH_FORCE_ALIGN_CHECK :
+ * This is a minor performance trick, only useful with lots of very small keys.
+ * It means : check for aligned/unaligned input.
+ * The check costs one initial branch per hash;
+ * set it to 0 when the input is guaranteed to be aligned,
+ * or when alignment doesn't matter for performance.
+ */
+#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
+# if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
+# define XXH_FORCE_ALIGN_CHECK 0
+# else
+# define XXH_FORCE_ALIGN_CHECK 1
+# endif
+#endif
+
+
+/* *************************************
+* Includes & Memory related functions
+***************************************/
+/*! Modify the local functions below should you wish to use some other memory routines
+* for malloc(), free() */
+#include <stdlib.h>
+static void* XXH_malloc(size_t s) { return malloc(s); }
+static void XXH_free (void* p) { free(p); }
+/*! and for memcpy() */
+#include <string.h>
+static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
+
+#include <assert.h> /* assert */
+
+#define XXH_STATIC_LINKING_ONLY
+#include "xxhash.h"
+
+
+/* *************************************
+* Compiler Specific Options
+***************************************/
+#ifdef _MSC_VER /* Visual Studio */
+# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
+# define XXH_FORCE_INLINE static __forceinline
+# define XXH_NO_INLINE static __declspec(noinline)
+#else
+# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
+# ifdef __GNUC__
+# define XXH_FORCE_INLINE static inline __attribute__((always_inline))
+# define XXH_NO_INLINE static __attribute__((noinline))
+# else
+# define XXH_FORCE_INLINE static inline
+# define XXH_NO_INLINE static
+# endif
+# else
+# define XXH_FORCE_INLINE static
+# define XXH_NO_INLINE static
+# endif /* __STDC_VERSION__ */
+#endif
+
+
+/* *************************************
+* Basic Types
+***************************************/
+#ifndef MEM_MODULE
+# if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+ typedef uint8_t BYTE;
+ typedef uint16_t U16;
+ typedef uint32_t U32;
+# else
+ typedef unsigned char BYTE;
+ typedef unsigned short U16;
+ typedef unsigned int U32;
+# endif
+#endif
+
+
+/* === Memory access === */
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; } __attribute__((packed)) unalign;
+static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+static U32 XXH_read32(const void* memPtr)
+{
+ U32 val;
+ memcpy(&val, memPtr, sizeof(val));
+ return val;
+}
+
+#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+
+/* === Endianess === */
+typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
+
+/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
+#ifndef XXH_CPU_LITTLE_ENDIAN
+static int XXH_isLittleEndian(void)
+{
+ const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */
+ return one.c[0];
+}
+# define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian()
+#endif
+
+
+
+
+/* ****************************************
+* Compiler-specific Functions and Macros
+******************************************/
+#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
+
+/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
+#if defined(_MSC_VER)
+# define XXH_rotl32(x,r) _rotl(x,r)
+# define XXH_rotl64(x,r) _rotl64(x,r)
+#else
+# define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r))))
+# define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r))))
+#endif
+
+#if defined(_MSC_VER) /* Visual Studio */
+# define XXH_swap32 _byteswap_ulong
+#elif XXH_GCC_VERSION >= 403
+# define XXH_swap32 __builtin_bswap32
+#else
+static U32 XXH_swap32 (U32 x)
+{
+ return ((x << 24) & 0xff000000 ) |
+ ((x << 8) & 0x00ff0000 ) |
+ ((x >> 8) & 0x0000ff00 ) |
+ ((x >> 24) & 0x000000ff );
+}
+#endif
+
+
+/* ***************************
+* Memory reads
+*****************************/
+typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
+
+XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
+}
+
+static U32 XXH_readBE32(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
+}
+
+XXH_FORCE_INLINE U32
+XXH_readLE32_align(const void* ptr, XXH_alignment align)
+{
+ if (align==XXH_unaligned) {
+ return XXH_readLE32(ptr);
+ } else {
+ return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
+ }
+}
+
+
+/* *************************************
+* Macros
+***************************************/
+#define XXH_STATIC_ASSERT(c) { enum { XXH_sa = 1/(int)(!!(c)) }; } /* use after variable declarations */
+XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }
+
+
+/* *******************************************************************
+* 32-bit hash functions
+*********************************************************************/
+static const U32 PRIME32_1 = 2654435761U; /* 0b10011110001101110111100110110001 */
+static const U32 PRIME32_2 = 2246822519U; /* 0b10000101111010111100101001110111 */
+static const U32 PRIME32_3 = 3266489917U; /* 0b11000010101100101010111000111101 */
+static const U32 PRIME32_4 = 668265263U; /* 0b00100111110101001110101100101111 */
+static const U32 PRIME32_5 = 374761393U; /* 0b00010110010101100110011110110001 */
+
+static U32 XXH32_round(U32 acc, U32 input)
+{
+ acc += input * PRIME32_2;
+ acc = XXH_rotl32(acc, 13);
+ acc *= PRIME32_1;
+#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE)
+ /* UGLY HACK:
+ * This inline assembly hack forces acc into a normal register. This is the
+ * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop
+ * (pragmas and attributes don't work for some resason) without globally
+ * disabling SSE4.1.
+ *
+ * The reason we want to avoid vectorization is because despite working on
+ * 4 integers at a time, there are multiple factors slowing XXH32 down on
+ * SSE4:
+ * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!)
+ * making it slightly slower to multiply four integers at once compared to four
+ * integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is
+ * still not worth it to go into SSE just to multiply unless doing a long operation.
+ *
+ * - Four instructions are required to rotate,
+ * movqda tmp, v // not required with VEX encoding
+ * pslld tmp, 13 // tmp <<= 13
+ * psrld v, 19 // x >>= 19
+ * por v, tmp // x |= tmp
+ * compared to one for scalar:
+ * roll v, 13 // reliably fast across the board
+ * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason
+ *
+ * - Instruction level parallelism is actually more beneficial here because the
+ * SIMD actually serializes this operation: While v1 is rotating, v2 can load data,
+ * while v3 can multiply. SSE forces them to operate together.
+ *
+ * How this hack works:
+ * __asm__("" // Declare an assembly block but don't declare any instructions
+ * : // However, as an Input/Output Operand,
+ * "+r" // constrain a read/write operand (+) as a general purpose register (r).
+ * (acc) // and set acc as the operand
+ * );
+ *
+ * Because of the 'r', the compiler has promised that seed will be in a
+ * general purpose register and the '+' says that it will be 'read/write',
+ * so it has to assume it has changed. It is like volatile without all the
+ * loads and stores.
+ *
+ * Since the argument has to be in a normal register (not an SSE register),
+ * each time XXH32_round is called, it is impossible to vectorize. */
+ __asm__("" : "+r" (acc));
+#endif
+ return acc;
+}
+
+/* mix all bits */
+static U32 XXH32_avalanche(U32 h32)
+{
+ h32 ^= h32 >> 15;
+ h32 *= PRIME32_2;
+ h32 ^= h32 >> 13;
+ h32 *= PRIME32_3;
+ h32 ^= h32 >> 16;
+ return(h32);
+}
+
+#define XXH_get32bits(p) XXH_readLE32_align(p, align)
+
+static U32
+XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align)
+
+{
+ const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1 \
+ h32 += (*p++) * PRIME32_5; \
+ h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
+
+#define PROCESS4 \
+ h32 += XXH_get32bits(p) * PRIME32_3; \
+ p+=4; \
+ h32 = XXH_rotl32(h32, 17) * PRIME32_4 ;
+
+ switch(len&15) /* or switch(bEnd - p) */
+ {
+ case 12: PROCESS4;
+ /* fallthrough */
+ case 8: PROCESS4;
+ /* fallthrough */
+ case 4: PROCESS4;
+ return XXH32_avalanche(h32);
+
+ case 13: PROCESS4;
+ /* fallthrough */
+ case 9: PROCESS4;
+ /* fallthrough */
+ case 5: PROCESS4;
+ PROCESS1;
+ return XXH32_avalanche(h32);
+
+ case 14: PROCESS4;
+ /* fallthrough */
+ case 10: PROCESS4;
+ /* fallthrough */
+ case 6: PROCESS4;
+ PROCESS1;
+ PROCESS1;
+ return XXH32_avalanche(h32);
+
+ case 15: PROCESS4;
+ /* fallthrough */
+ case 11: PROCESS4;
+ /* fallthrough */
+ case 7: PROCESS4;
+ /* fallthrough */
+ case 3: PROCESS1;
+ /* fallthrough */
+ case 2: PROCESS1;
+ /* fallthrough */
+ case 1: PROCESS1;
+ /* fallthrough */
+ case 0: return XXH32_avalanche(h32);
+ }
+ assert(0);
+ return h32; /* reaching this point is deemed impossible */
+}
+
+XXH_FORCE_INLINE U32
+XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)input;
+ const BYTE* bEnd = p + len;
+ U32 h32;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ if (p==NULL) {
+ len=0;
+ bEnd=p=(const BYTE*)(size_t)16;
+ }
+#endif
+
+ if (len>=16) {
+ const BYTE* const limit = bEnd - 15;
+ U32 v1 = seed + PRIME32_1 + PRIME32_2;
+ U32 v2 = seed + PRIME32_2;
+ U32 v3 = seed + 0;
+ U32 v4 = seed - PRIME32_1;
+
+ do {
+ v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
+ v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
+ v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
+ v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
+ } while (p < limit);
+
+ h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7)
+ + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
+ } else {
+ h32 = seed + PRIME32_5;
+ }
+
+ h32 += (U32)len;
+
+ return XXH32_finalize(h32, p, len&15, align);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
+{
+#if 0
+ /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+ XXH32_state_t state;
+ XXH32_reset(&state, seed);
+ XXH32_update(&state, input, len);
+ return XXH32_digest(&state);
+
+#else
+
+ if (XXH_FORCE_ALIGN_CHECK) {
+ if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */
+ return XXH32_endian_align(input, len, seed, XXH_aligned);
+ } }
+
+ return XXH32_endian_align(input, len, seed, XXH_unaligned);
+#endif
+}
+
+
+
+/*====== Hash streaming ======*/
+
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
+{
+ return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
+{
+ XXH_free(statePtr);
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState)
+{
+ memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
+{
+ XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+ memset(&state, 0, sizeof(state));
+ state.v1 = seed + PRIME32_1 + PRIME32_2;
+ state.v2 = seed + PRIME32_2;
+ state.v3 = seed + 0;
+ state.v4 = seed - PRIME32_1;
+ /* do not write into reserved, planned to be removed in a future version */
+ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API XXH_errorcode
+XXH32_update(XXH32_state_t* state, const void* input, size_t len)
+{
+ if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ return XXH_OK;
+#else
+ return XXH_ERROR;
+#endif
+
+ { const BYTE* p = (const BYTE*)input;
+ const BYTE* const bEnd = p + len;
+
+ state->total_len_32 += (XXH32_hash_t)len;
+ state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16));
+
+ if (state->memsize + len < 16) { /* fill in tmp buffer */
+ XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
+ state->memsize += (XXH32_hash_t)len;
+ return XXH_OK;
+ }
+
+ if (state->memsize) { /* some data left from previous update */
+ XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
+ { const U32* p32 = state->mem32;
+ state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++;
+ state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++;
+ state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++;
+ state->v4 = XXH32_round(state->v4, XXH_readLE32(p32));
+ }
+ p += 16-state->memsize;
+ state->memsize = 0;
+ }
+
+ if (p <= bEnd-16) {
+ const BYTE* const limit = bEnd - 16;
+ U32 v1 = state->v1;
+ U32 v2 = state->v2;
+ U32 v3 = state->v3;
+ U32 v4 = state->v4;
+
+ do {
+ v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4;
+ v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4;
+ v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4;
+ v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4;
+ } while (p<=limit);
+
+ state->v1 = v1;
+ state->v2 = v2;
+ state->v3 = v3;
+ state->v4 = v4;
+ }
+
+ if (p < bEnd) {
+ XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
+ state->memsize = (unsigned)(bEnd-p);
+ }
+ }
+
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state)
+{
+ U32 h32;
+
+ if (state->large_len) {
+ h32 = XXH_rotl32(state->v1, 1)
+ + XXH_rotl32(state->v2, 7)
+ + XXH_rotl32(state->v3, 12)
+ + XXH_rotl32(state->v4, 18);
+ } else {
+ h32 = state->v3 /* == seed */ + PRIME32_5;
+ }
+
+ h32 += state->total_len_32;
+
+ return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned);
+}
+
+
+/*====== Canonical representation ======*/
+
+/*! Default XXH result types are basic unsigned 32 and 64 bits.
+* The canonical representation follows human-readable write convention, aka big-endian (large digits first).
+* These functions allow transformation of hash result into and from its canonical format.
+* This way, hash values can be written into a file or buffer, remaining comparable across different systems.
+*/
+
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
+{
+ XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
+ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
+ memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
+{
+ return XXH_readBE32(src);
+}
+
+
+#ifndef XXH_NO_LONG_LONG
+
+/* *******************************************************************
+* 64-bit hash functions
+*********************************************************************/
+
+/*====== Memory access ======*/
+
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+ typedef uint64_t U64;
+# else
+ /* if compiler doesn't support unsigned long long, replace by another 64-bit type */
+ typedef unsigned long long U64;
+# endif
+#endif
+
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64;
+static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+
+static U64 XXH_read64(const void* memPtr)
+{
+ U64 val;
+ memcpy(&val, memPtr, sizeof(val));
+ return val;
+}
+
+#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+#if defined(_MSC_VER) /* Visual Studio */
+# define XXH_swap64 _byteswap_uint64
+#elif XXH_GCC_VERSION >= 403
+# define XXH_swap64 __builtin_bswap64
+#else
+static U64 XXH_swap64 (U64 x)
+{
+ return ((x << 56) & 0xff00000000000000ULL) |
+ ((x << 40) & 0x00ff000000000000ULL) |
+ ((x << 24) & 0x0000ff0000000000ULL) |
+ ((x << 8) & 0x000000ff00000000ULL) |
+ ((x >> 8) & 0x00000000ff000000ULL) |
+ ((x >> 24) & 0x0000000000ff0000ULL) |
+ ((x >> 40) & 0x000000000000ff00ULL) |
+ ((x >> 56) & 0x00000000000000ffULL);
+}
+#endif
+
+XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
+}
+
+static U64 XXH_readBE64(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
+}
+
+XXH_FORCE_INLINE U64
+XXH_readLE64_align(const void* ptr, XXH_alignment align)
+{
+ if (align==XXH_unaligned)
+ return XXH_readLE64(ptr);
+ else
+ return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
+}
+
+
+/*====== xxh64 ======*/
+
+static const U64 PRIME64_1 = 11400714785074694791ULL; /* 0b1001111000110111011110011011000110000101111010111100101010000111 */
+static const U64 PRIME64_2 = 14029467366897019727ULL; /* 0b1100001010110010101011100011110100100111110101001110101101001111 */
+static const U64 PRIME64_3 = 1609587929392839161ULL; /* 0b0001011001010110011001111011000110011110001101110111100111111001 */
+static const U64 PRIME64_4 = 9650029242287828579ULL; /* 0b1000010111101011110010100111011111000010101100101010111001100011 */
+static const U64 PRIME64_5 = 2870177450012600261ULL; /* 0b0010011111010100111010110010111100010110010101100110011111000101 */
+
+static U64 XXH64_round(U64 acc, U64 input)
+{
+ acc += input * PRIME64_2;
+ acc = XXH_rotl64(acc, 31);
+ acc *= PRIME64_1;
+ return acc;
+}
+
+static U64 XXH64_mergeRound(U64 acc, U64 val)
+{
+ val = XXH64_round(0, val);
+ acc ^= val;
+ acc = acc * PRIME64_1 + PRIME64_4;
+ return acc;
+}
+
+static U64 XXH64_avalanche(U64 h64)
+{
+ h64 ^= h64 >> 33;
+ h64 *= PRIME64_2;
+ h64 ^= h64 >> 29;
+ h64 *= PRIME64_3;
+ h64 ^= h64 >> 32;
+ return h64;
+}
+
+
+#define XXH_get64bits(p) XXH_readLE64_align(p, align)
+
+static U64
+XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1_64 \
+ h64 ^= (*p++) * PRIME64_5; \
+ h64 = XXH_rotl64(h64, 11) * PRIME64_1;
+
+#define PROCESS4_64 \
+ h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \
+ p+=4; \
+ h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
+
+#define PROCESS8_64 { \
+ U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \
+ p+=8; \
+ h64 ^= k1; \
+ h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \
+}
+
+ switch(len&31) {
+ case 24: PROCESS8_64;
+ /* fallthrough */
+ case 16: PROCESS8_64;
+ /* fallthrough */
+ case 8: PROCESS8_64;
+ return XXH64_avalanche(h64);
+
+ case 28: PROCESS8_64;
+ /* fallthrough */
+ case 20: PROCESS8_64;
+ /* fallthrough */
+ case 12: PROCESS8_64;
+ /* fallthrough */
+ case 4: PROCESS4_64;
+ return XXH64_avalanche(h64);
+
+ case 25: PROCESS8_64;
+ /* fallthrough */
+ case 17: PROCESS8_64;
+ /* fallthrough */
+ case 9: PROCESS8_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 29: PROCESS8_64;
+ /* fallthrough */
+ case 21: PROCESS8_64;
+ /* fallthrough */
+ case 13: PROCESS8_64;
+ /* fallthrough */
+ case 5: PROCESS4_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 26: PROCESS8_64;
+ /* fallthrough */
+ case 18: PROCESS8_64;
+ /* fallthrough */
+ case 10: PROCESS8_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 30: PROCESS8_64;
+ /* fallthrough */
+ case 22: PROCESS8_64;
+ /* fallthrough */
+ case 14: PROCESS8_64;
+ /* fallthrough */
+ case 6: PROCESS4_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 27: PROCESS8_64;
+ /* fallthrough */
+ case 19: PROCESS8_64;
+ /* fallthrough */
+ case 11: PROCESS8_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 31: PROCESS8_64;
+ /* fallthrough */
+ case 23: PROCESS8_64;
+ /* fallthrough */
+ case 15: PROCESS8_64;
+ /* fallthrough */
+ case 7: PROCESS4_64;
+ /* fallthrough */
+ case 3: PROCESS1_64;
+ /* fallthrough */
+ case 2: PROCESS1_64;
+ /* fallthrough */
+ case 1: PROCESS1_64;
+ /* fallthrough */
+ case 0: return XXH64_avalanche(h64);
+ }
+
+ /* impossible to reach */
+ assert(0);
+ return 0; /* unreachable, but some compilers complain without it */
+}
+
+XXH_FORCE_INLINE U64
+XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)input;
+ const BYTE* bEnd = p + len;
+ U64 h64;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ if (p==NULL) {
+ len=0;
+ bEnd=p=(const BYTE*)(size_t)32;
+ }
+#endif
+
+ if (len>=32) {
+ const BYTE* const limit = bEnd - 32;
+ U64 v1 = seed + PRIME64_1 + PRIME64_2;
+ U64 v2 = seed + PRIME64_2;
+ U64 v3 = seed + 0;
+ U64 v4 = seed - PRIME64_1;
+
+ do {
+ v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
+ v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
+ v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
+ v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
+ } while (p<=limit);
+
+ h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+ h64 = XXH64_mergeRound(h64, v1);
+ h64 = XXH64_mergeRound(h64, v2);
+ h64 = XXH64_mergeRound(h64, v3);
+ h64 = XXH64_mergeRound(h64, v4);
+
+ } else {
+ h64 = seed + PRIME64_5;
+ }
+
+ h64 += (U64) len;
+
+ return XXH64_finalize(h64, p, len, align);
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
+{
+#if 0
+ /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+ XXH64_state_t state;
+ XXH64_reset(&state, seed);
+ XXH64_update(&state, input, len);
+ return XXH64_digest(&state);
+
+#else
+
+ if (XXH_FORCE_ALIGN_CHECK) {
+ if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */
+ return XXH64_endian_align(input, len, seed, XXH_aligned);
+ } }
+
+ return XXH64_endian_align(input, len, seed, XXH_unaligned);
+
+#endif
+}
+
+/*====== Hash Streaming ======*/
+
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
+{
+ return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
+{
+ XXH_free(statePtr);
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState)
+{
+ memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
+{
+ XXH64_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+ memset(&state, 0, sizeof(state));
+ state.v1 = seed + PRIME64_1 + PRIME64_2;
+ state.v2 = seed + PRIME64_2;
+ state.v3 = seed + 0;
+ state.v4 = seed - PRIME64_1;
+ /* do not write into reserved, planned to be removed in a future version */
+ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API XXH_errorcode
+XXH64_update (XXH64_state_t* state, const void* input, size_t len)
+{
+ if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ return XXH_OK;
+#else
+ return XXH_ERROR;
+#endif
+
+ { const BYTE* p = (const BYTE*)input;
+ const BYTE* const bEnd = p + len;
+
+ state->total_len += len;
+
+ if (state->memsize + len < 32) { /* fill in tmp buffer */
+ XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+ state->memsize += (U32)len;
+ return XXH_OK;
+ }
+
+ if (state->memsize) { /* tmp buffer is full */
+ XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
+ state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0));
+ state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1));
+ state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2));
+ state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3));
+ p += 32-state->memsize;
+ state->memsize = 0;
+ }
+
+ if (p+32 <= bEnd) {
+ const BYTE* const limit = bEnd - 32;
+ U64 v1 = state->v1;
+ U64 v2 = state->v2;
+ U64 v3 = state->v3;
+ U64 v4 = state->v4;
+
+ do {
+ v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8;
+ v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8;
+ v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8;
+ v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8;
+ } while (p<=limit);
+
+ state->v1 = v1;
+ state->v2 = v2;
+ state->v3 = v3;
+ state->v4 = v4;
+ }
+
+ if (p < bEnd) {
+ XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
+ state->memsize = (unsigned)(bEnd-p);
+ }
+ }
+
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state)
+{
+ U64 h64;
+
+ if (state->total_len >= 32) {
+ U64 const v1 = state->v1;
+ U64 const v2 = state->v2;
+ U64 const v3 = state->v3;
+ U64 const v4 = state->v4;
+
+ h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+ h64 = XXH64_mergeRound(h64, v1);
+ h64 = XXH64_mergeRound(h64, v2);
+ h64 = XXH64_mergeRound(h64, v3);
+ h64 = XXH64_mergeRound(h64, v4);
+ } else {
+ h64 = state->v3 /*seed*/ + PRIME64_5;
+ }
+
+ h64 += (U64) state->total_len;
+
+ return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned);
+}
+
+
+/*====== Canonical representation ======*/
+
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
+{
+ XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
+ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
+ memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
+{
+ return XXH_readBE64(src);
+}
+
+
+
+/* *********************************************************************
+* XXH3
+* New generation hash designed for speed on small keys and vectorization
+************************************************************************ */
+
+/* #include "xxh3.h" */
+
+
+
+#endif /* XXH_NO_LONG_LONG */
diff --git a/crypto/xxhash.h b/crypto/xxhash.h
new file mode 100644
index 000000000000..9ee05e5dc146
--- /dev/null
+++ b/crypto/xxhash.h
@@ -0,0 +1,445 @@
+/*
+ xxHash - Extremely Fast Hash algorithm
+ Header File
+ Copyright (C) 2012-2016, Yann Collet.
+
+ BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with the
+ distribution.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ You can contact the author at :
+ - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* Notice extracted from xxHash homepage :
+
+xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
+It also successfully passes all tests from the SMHasher suite.
+
+Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
+
+Name Speed Q.Score Author
+xxHash 5.4 GB/s 10
+CrapWow 3.2 GB/s 2 Andrew
+MumurHash 3a 2.7 GB/s 10 Austin Appleby
+SpookyHash 2.0 GB/s 10 Bob Jenkins
+SBox 1.4 GB/s 9 Bret Mulvey
+Lookup3 1.2 GB/s 9 Bob Jenkins
+SuperFastHash 1.2 GB/s 1 Paul Hsieh
+CityHash64 1.05 GB/s 10 Pike & Alakuijala
+FNV 0.55 GB/s 5 Fowler, Noll, Vo
+CRC32 0.43 GB/s 9
+MD5-32 0.33 GB/s 10 Ronald L. Rivest
+SHA1-32 0.28 GB/s 10
+
+Q.Score is a measure of quality of the hash function.
+It depends on successfully passing SMHasher test set.
+10 is a perfect score.
+
+A 64-bit version, named XXH64, is available since r35.
+It offers much better speed, but for 64-bit applications only.
+Name Speed on 64 bits Speed on 32 bits
+XXH64 13.8 GB/s 1.9 GB/s
+XXH32 6.8 GB/s 6.0 GB/s
+*/
+
+#ifndef XXHASH_H_5627135585666179
+#define XXHASH_H_5627135585666179 1
+
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+
+/* ****************************
+* Definitions
+******************************/
+#include <stddef.h> /* size_t */
+typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
+
+
+/* ****************************
+ * API modifier
+ ******************************/
+/** XXH_INLINE_ALL (and XXH_PRIVATE_API)
+ * This is useful to include xxhash functions in `static` mode
+ * in order to inline them, and remove their symbol from the public list.
+ * Inlining can offer dramatic performance improvement on small keys.
+ * Methodology :
+ * #define XXH_INLINE_ALL
+ * #include "xxhash.h"
+ * `xxhash.c` is automatically included.
+ * It's not useful to compile and link it as a separate module.
+ */
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+# ifndef XXH_STATIC_LINKING_ONLY
+# define XXH_STATIC_LINKING_ONLY
+# endif
+# if defined(__GNUC__)
+# define XXH_PUBLIC_API static __inline __attribute__((unused))
+# elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
+# define XXH_PUBLIC_API static inline
+# elif defined(_MSC_VER)
+# define XXH_PUBLIC_API static __inline
+# else
+ /* this version may generate warnings for unused static functions */
+# define XXH_PUBLIC_API static
+# endif
+#else
+# if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT))
+# ifdef XXH_EXPORT
+# define XXH_PUBLIC_API __declspec(dllexport)
+# elif XXH_IMPORT
+# define XXH_PUBLIC_API __declspec(dllimport)
+# endif
+# else
+# define XXH_PUBLIC_API /* do nothing */
+# endif
+#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */
+
+/*! XXH_NAMESPACE, aka Namespace Emulation :
+ *
+ * If you want to include _and expose_ xxHash functions from within your own library,
+ * but also want to avoid symbol collisions with other libraries which may also include xxHash,
+ *
+ * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
+ * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values).
+ *
+ * Note that no change is required within the calling program as long as it includes `xxhash.h` :
+ * regular symbol name will be automatically translated by this header.
+ */
+#ifdef XXH_NAMESPACE
+# define XXH_CAT(A,B) A##B
+# define XXH_NAME2(A,B) XXH_CAT(A,B)
+# define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
+# define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
+# define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
+# define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
+# define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
+# define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
+# define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
+# define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
+# define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
+# define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
+# define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
+# define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
+# define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
+# define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
+# define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
+# define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
+# define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
+# define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
+# define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
+#endif
+
+
+/* *************************************
+* Version
+***************************************/
+#define XXH_VERSION_MAJOR 0
+#define XXH_VERSION_MINOR 7
+#define XXH_VERSION_RELEASE 0
+#define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
+XXH_PUBLIC_API unsigned XXH_versionNumber (void);
+
+
+/*-**********************************************************************
+* 32-bit hash
+************************************************************************/
+typedef unsigned int XXH32_hash_t;
+
+/*! XXH32() :
+ Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input".
+ The memory between input & input+length must be valid (allocated and read-accessible).
+ "seed" can be used to alter the result predictably.
+ Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */
+XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
+
+/*====== Streaming ======*/
+typedef struct XXH32_state_s XXH32_state_t; /* incomplete type */
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr);
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed);
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr);
+
+/*
+ * Streaming functions generate the xxHash of an input provided in multiple segments.
+ * Note that, for small input, they are slower than single-call functions, due to state management.
+ * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized.
+ *
+ * XXH state must first be allocated, using XXH*_createState() .
+ *
+ * Start a new hash by initializing state with a seed, using XXH*_reset().
+ *
+ * Then, feed the hash state by calling XXH*_update() as many times as necessary.
+ * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
+ *
+ * Finally, a hash value can be produced anytime, by using XXH*_digest().
+ * This function returns the nn-bits hash as an int or long long.
+ *
+ * It's still possible to continue inserting input into the hash state after a digest,
+ * and generate some new hashes later on, by calling again XXH*_digest().
+ *
+ * When done, free XXH state space if it was allocated dynamically.
+ */
+
+/*====== Canonical representation ======*/
+
+typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
+
+/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
+ * The canonical representation uses human-readable write convention, aka big-endian (large digits first).
+ * These functions allow transformation of hash result into and from its canonical format.
+ * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
+ */
+
+
+#ifndef XXH_NO_LONG_LONG
+/*-**********************************************************************
+* 64-bit hash
+************************************************************************/
+typedef unsigned long long XXH64_hash_t;
+
+/*! XXH64() :
+ Calculate the 64-bit hash of sequence of length "len" stored at memory address "input".
+ "seed" can be used to alter the result predictably.
+ This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark).
+*/
+XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);
+
+/*====== Streaming ======*/
+typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr);
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed);
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr);
+
+/*====== Canonical representation ======*/
+typedef struct { unsigned char digest[8]; } XXH64_canonical_t;
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);
+
+
+#endif /* XXH_NO_LONG_LONG */
+
+
+
+#ifdef XXH_STATIC_LINKING_ONLY
+
+/* ================================================================================================
+ This section contains declarations which are not guaranteed to remain stable.
+ They may change in future versions, becoming incompatible with a different version of the library.
+ These declarations should only be used with static linking.
+ Never use them in association with dynamic linking !
+=================================================================================================== */
+
+/* These definitions are only present to allow
+ * static allocation of XXH state, on stack or in a struct for example.
+ * Never **ever** use members directly. */
+
+#if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+
+struct XXH32_state_s {
+ uint32_t total_len_32;
+ uint32_t large_len;
+ uint32_t v1;
+ uint32_t v2;
+ uint32_t v3;
+ uint32_t v4;
+ uint32_t mem32[4];
+ uint32_t memsize;
+ uint32_t reserved; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH32_state_t */
+
+struct XXH64_state_s {
+ uint64_t total_len;
+ uint64_t v1;
+ uint64_t v2;
+ uint64_t v3;
+ uint64_t v4;
+ uint64_t mem64[4];
+ uint32_t memsize;
+ uint32_t reserved[2]; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH64_state_t */
+
+# else
+
+struct XXH32_state_s {
+ XXH32_hash_t total_len_32;
+ XXH32_hash_t large_len;
+ XXH32_hash_t v1;
+ XXH32_hash_t v2;
+ XXH32_hash_t v3;
+ XXH32_hash_t v4;
+ XXH32_hash_t mem32[4];
+ XXH32_hash_t memsize;
+ XXH32_hash_t reserved; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH32_state_t */
+
+# ifndef XXH_NO_LONG_LONG /* remove 64-bit support */
+struct XXH64_state_s {
+ XXH64_hash_t total_len;
+ XXH64_hash_t v1;
+ XXH64_hash_t v2;
+ XXH64_hash_t v3;
+ XXH64_hash_t v4;
+ XXH64_hash_t mem64[4];
+ XXH32_hash_t memsize;
+ XXH32_hash_t reserved[2]; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH64_state_t */
+# endif
+
+# endif
+
+
+/*-**********************************************************************
+* XXH3
+* New experimental hash
+************************************************************************/
+#ifndef XXH_NO_LONG_LONG
+
+
+/* ============================================
+ * XXH3 is a new hash algorithm,
+ * featuring vastly improved speed performance
+ * for both small and large inputs.
+ * A full speed analysis will be published,
+ * it requires a lot more space than this comment can handle.
+ * In general, expect XXH3 to run about ~2x faster on large inputs,
+ * and >3x faster on small ones, though exact difference depend on platform.
+ *
+ * The algorithm is portable, will generate the same hash on all platforms.
+ * It benefits greatly from vectorization units, but does not require it.
+ *
+ * XXH3 offers 2 variants, _64bits and _128bits.
+ * When only 64 bits are needed, prefer calling the _64bits variant :
+ * it reduces the amount of mixing, resulting in faster speed on small inputs.
+ * It's also generally simpler to manipulate a scalar type than a struct.
+ * Note : the low 64-bit field of the _128bits variant is the same as _64bits result.
+ *
+ * The XXH3 algorithm is still considered experimental.
+ * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use.
+ * While labelled experimental, the produced result can still change between versions.
+ *
+ * The API currently supports one-shot hashing only.
+ * The full version will include streaming capability, and canonical representation
+ * Long term optional feature may include custom secret keys, and secret key generation.
+ *
+ * There are still a number of opened questions that community can influence during the experimental period.
+ * I'm trying to list a few of them below, though don't consider this list as complete.
+ *
+ * - 128-bits output type : currently defined as a structure of 2 64-bits fields.
+ * That's because 128-bit values do not exist in C standard.
+ * Note that it means that, at byte level, result is not identical depending on endianess.
+ * However, at field level, they are identical on all platforms.
+ * The canonical representation will solve the issue of identical byte-level representation across platforms,
+ * which is necessary for serialization.
+ * Would there be a better representation for a 128-bit hash result ?
+ * Are the names of the inner 64-bit fields important ? Should they be changed ?
+ *
+ * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian).
+ * What should it be for the 128-bit variant ?
+ * Since it's no longer a scalar value, big-endian representation is no longer an obvious choice.
+ * One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian)
+ * Another one : represent it in the same order as natural order in the struct for little-endian platforms.
+ * Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms.
+ *
+ * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct.
+ * Granted, it's not terribly difficult to create a comparator, but it's still a workload.
+ * Would it be beneficial to declare and define a comparator function for XXH128_hash_t ?
+ * Are there other operations on XXH128_hash_t which would be desirable ?
+ *
+ * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits.
+ * This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases.
+ * But it might influence the design of XXH128_hash_t, in ways which may block other possibilities.
+ * Good idea, bad idea ?
+ *
+ * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant.
+ * It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash.
+ * Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value.
+ * It would either replace current choice, or add a new one.
+ * Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`).
+ * If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ?
+ *
+ * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed.
+ * This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty.
+ * Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64).
+ * But is it still fine to return the seed when the seed is non-zero ?
+ * Are there use case which would depend on this behavior, or would prefer a mixing of the seed ?
+ */
+
+#ifdef XXH_NAMESPACE
+# define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128)
+# define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits)
+# define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed)
+# define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits)
+# define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed)
+#endif
+
+
+typedef struct {
+ XXH64_hash_t low64;
+ XXH64_hash_t high64;
+} XXH128_hash_t;
+
+XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed);
+
+/* note : variants without seed produce same result as variant with seed == 0 */
+XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed); /* == XXH128() */
+
+
+#endif /* XXH_NO_LONG_LONG */
+
+
+/*-**********************************************************************
+* XXH_INLINE_ALL
+************************************************************************/
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+# include "xxhash.c" /* include xxhash function bodies as `static`, for inlining */
+#endif
+
+
+
+#endif /* XXH_STATIC_LINKING_ONLY */
+
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* XXHASH_H_5627135585666179 */
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (8 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 09/12] btrfs-progs: add xxhash sources Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-04 8:31 ` Nikolay Borisov
2019-09-24 14:23 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
` (2 subsequent siblings)
12 siblings, 2 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
Add xxhash64 as another checksumming algorithm.
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Changes to v3:
- Fix usage of is_valid_csum_type() (Nikolay)
- Remove unrelated whitespace change (Nikolay)
Changes to v2:
- Integrated comments from Nikolay
---
Makefile | 3 ++-
cmds/inspect-dump-super.c | 24 +++++++++++++++---------
convert/common.c | 2 +-
convert/main.c | 2 +-
crypto/hash.c | 16 ++++++++++++++++
crypto/hash.h | 10 ++++++++++
ctree.h | 14 ++++++++++----
disk-io.c | 7 +++++--
image/main.c | 5 +++--
mkfs/common.c | 14 +++++++-------
mkfs/main.c | 6 +++++-
11 files changed, 75 insertions(+), 28 deletions(-)
create mode 100644 crypto/hash.c
create mode 100644 crypto/hash.h
diff --git a/Makefile b/Makefile
index 370e0c37ff65..45530749e2b9 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
kernel-lib/crc32c.o common/messages.o \
- uuid-tree.o utils-lib.o common/rbtree-utils.o
+ uuid-tree.o utils-lib.o common/rbtree-utils.o \
+ crypto/hash.o crypto/xxhash.o
libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 58bf82b0bbd3..f9f38751f429 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
super_flags_num, BTRFS_SUPER_FLAG_SUPP);
}
+static bool is_valid_csum_type(u16 csum_type)
+{
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ case BTRFS_CSUM_TYPE_XXHASH:
+ return true;
+ default:
+ return false;
+ }
+}
+
static void dump_superblock(struct btrfs_super_block *sb, int full)
{
int i;
@@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
csum_type = btrfs_super_csum_type(sb);
csum_size = BTRFS_CSUM_SIZE;
printf("csum_type\t\t%hu (", csum_type);
- if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+ if (!is_valid_csum_type(csum_type)) {
printf("INVALID");
} else {
- if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
- printf("crc32c");
- csum_size = btrfs_csum_sizes[csum_type];
- } else {
- printf("unknown");
- }
+ printf("%s", btrfs_csums[csum_type].name);
+ csum_size = btrfs_csums[csum_type].size;
}
printf(")\n");
printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
@@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
printf("csum\t\t\t0x");
for (i = 0, p = sb->csum; i < csum_size; i++)
printf("%02x", p[i]);
- if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
- csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
+ if (!is_valid_csum_type(csum_type))
printf(" [UNKNOWN CSUM TYPE OR SIZE]");
else if (check_csum_sblock(sb, csum_size, csum_type))
printf(" [match]");
diff --git a/convert/common.c b/convert/common.c
index 2e2318a5863e..5dd1a2644bf6 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
{
int ret;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
/* Temporary extent buffer is always mapped 1:1 on disk */
diff --git a/convert/main.c b/convert/main.c
index 5e6b12431f59..5eb2a59fb68a 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+ csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
btrfs_super_csum_type(super));
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
BTRFS_SUPER_INFO_OFFSET);
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..fda7fc4e9f23
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,16 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+ XXH64_hash_t hash;
+
+ hash = XXH64(buf, length, 0);
+ /* NOTE: we're not taking the canonical form here but the plain hash to
+ * be compatible with the kernel implementation!
+ */
+ memcpy(out, &hash, 8);
+
+ return 0;
+}
+
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX 32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/ctree.h b/ctree.h
index 870d9f4948de..4ded8161d149 100644
--- a/ctree.h
+++ b/ctree.h
@@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
/* csum types */
enum btrfs_csum_type {
BTRFS_CSUM_TYPE_CRC32 = 0,
+ BTRFS_CSUM_TYPE_XXHASH = 1,
};
-/* four bytes for CRC32 */
-static int btrfs_csum_sizes[] = { 4 };
+static struct btrfs_csum {
+ u16 size;
+ const char *name;
+} btrfs_csums[] = {
+ [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+ [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
+};
#define BTRFS_EMPTY_DIR_SIZE 0
@@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
{
int t = btrfs_super_csum_type(s);
- BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
- return btrfs_csum_sizes[t];
+ BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
+ return btrfs_csums[t].size;
}
static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
diff --git a/disk-io.c b/disk-io.c
index 810c2e14294a..ce0b746f4db9 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
#include "print-tree.h"
#include "common/rbtree-utils.h"
#include "common/device-scan.h"
+#include "crypto/hash.h"
/* specified errno for check_tree_block */
#define BTRFS_BAD_BYTENR (-1)
@@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
crc = crc32c(crc, data, len);
put_unaligned_le32(~crc, out);
return 0;
+ case BTRFS_CSUM_TYPE_XXHASH:
+ return hash_xxhash(data, len, out);
default:
fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
ASSERT(0);
@@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
}
csum_type = btrfs_super_csum_type(sb);
- if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+ if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
error("unsupported checksum algorithm %u", csum_type);
return -EIO;
}
- csum_size = btrfs_csum_sizes[csum_type];
+ csum_size = btrfs_csums[csum_type].size;
btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
diff --git a/image/main.c b/image/main.c
index 0c8ffede56f5..1265152cf524 100644
--- a/image/main.c
+++ b/image/main.c
@@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
static void csum_block(u8 *buf, size_t len)
{
- u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
+ u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
+ u8 result[csum_size];
u32 crc = ~(u32)0;
crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
put_unaligned_le32(~crc, result);
- memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
+ memcpy(buf, result, csum_size);
}
static int has_name(struct btrfs_key *key)
diff --git a/mkfs/common.c b/mkfs/common.c
index 4a417bd7a306..939be5eb2dc2 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
/* write back root tree */
@@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
@@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
@@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
@@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
@@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
@@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
diff --git a/mkfs/main.c b/mkfs/main.c
index e96cbc5399a2..64806dac7706 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
{
if (strcasecmp(s, "crc32c") == 0) {
return BTRFS_CSUM_TYPE_CRC32;
+ } else if (strcasecmp(s, "xxhash64") == 0 ||
+ strcasecmp(s, "xxhash") == 0) {
+ return BTRFS_CSUM_TYPE_XXHASH;
} else {
error("unknown csum type %s", s);
exit(1);
@@ -1376,7 +1379,8 @@ raid_groups:
pretty_size(allocation.system));
printf("SSD detected: %s\n", ssd ? "yes" : "no");
btrfs_parse_features_to_string(features_buf, features);
- printf("Incompat features: %s", features_buf);
+ printf("Incompat features: %s\n", features_buf);
+ printf("Checksum: %s", btrfs_csums[csum_type].name);
printf("\n");
list_all_devices(root);
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
@ 2019-09-04 8:31 ` Nikolay Borisov
2019-09-24 14:23 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: Nikolay Borisov @ 2019-09-04 8:31 UTC (permalink / raw)
To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist
On 3.09.19 г. 18:00 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
>
> Add xxhash64 as another checksumming algorithm.
>
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
>
> ---
> Changes to v3:
> - Fix usage of is_valid_csum_type() (Nikolay)
> - Remove unrelated whitespace change (Nikolay)
>
> Changes to v2:
> - Integrated comments from Nikolay
> ---
> Makefile | 3 ++-
> cmds/inspect-dump-super.c | 24 +++++++++++++++---------
> convert/common.c | 2 +-
> convert/main.c | 2 +-
> crypto/hash.c | 16 ++++++++++++++++
> crypto/hash.h | 10 ++++++++++
> ctree.h | 14 ++++++++++----
> disk-io.c | 7 +++++--
> image/main.c | 5 +++--
> mkfs/common.c | 14 +++++++-------
> mkfs/main.c | 6 +++++-
> 11 files changed, 75 insertions(+), 28 deletions(-)
> create mode 100644 crypto/hash.c
> create mode 100644 crypto/hash.h
>
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> kernel-lib/crc32c.o common/messages.o \
> - uuid-tree.o utils-lib.o common/rbtree-utils.o
> + uuid-tree.o utils-lib.o common/rbtree-utils.o \
> + crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 58bf82b0bbd3..f9f38751f429 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
> super_flags_num, BTRFS_SUPER_FLAG_SUPP);
> }
>
> +static bool is_valid_csum_type(u16 csum_type)
> +{
> + switch (csum_type) {
> + case BTRFS_CSUM_TYPE_CRC32:
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> static void dump_superblock(struct btrfs_super_block *sb, int full)
> {
> int i;
> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> csum_type = btrfs_super_csum_type(sb);
> csum_size = BTRFS_CSUM_SIZE;
> printf("csum_type\t\t%hu (", csum_type);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (!is_valid_csum_type(csum_type)) {
> printf("INVALID");
> } else {
> - if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
> - printf("crc32c");
> - csum_size = btrfs_csum_sizes[csum_type];
> - } else {
> - printf("unknown");
> - }
> + printf("%s", btrfs_csums[csum_type].name);
> + csum_size = btrfs_csums[csum_type].size;
> }
> printf(")\n");
> printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
> @@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> printf("csum\t\t\t0x");
> for (i = 0, p = sb->csum; i < csum_size; i++)
> printf("%02x", p[i]);
> - if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> + if (!is_valid_csum_type(csum_type))
> printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> else if (check_csum_sblock(sb, csum_size, csum_type))
> printf(" [match]");
> diff --git a/convert/common.c b/convert/common.c
> index 2e2318a5863e..5dd1a2644bf6 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
> {
> int ret;
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* Temporary extent buffer is always mapped 1:1 on disk */
> diff --git a/convert/main.c b/convert/main.c
> index 5e6b12431f59..5eb2a59fb68a 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
> BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
> btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
> btrfs_super_csum_type(super));
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> BTRFS_SUPER_INFO_OFFSET);
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..fda7fc4e9f23
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> + XXH64_hash_t hash;
> +
> + hash = XXH64(buf, length, 0);
> + /* NOTE: we're not taking the canonical form here but the plain hash to
> + * be compatible with the kernel implementation!
> + */
> + memcpy(out, &hash, 8);
> +
> + return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX 32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.h b/ctree.h
> index 870d9f4948de..4ded8161d149 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
> /* csum types */
> enum btrfs_csum_type {
> BTRFS_CSUM_TYPE_CRC32 = 0,
> + BTRFS_CSUM_TYPE_XXHASH = 1,
> };
>
> -/* four bytes for CRC32 */
> -static int btrfs_csum_sizes[] = { 4 };
> +static struct btrfs_csum {
> + u16 size;
> + const char *name;
> +} btrfs_csums[] = {
> + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
> +};
>
> #define BTRFS_EMPTY_DIR_SIZE 0
>
> @@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
> static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
> {
> int t = btrfs_super_csum_type(s);
> - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
> - return btrfs_csum_sizes[t];
> + BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
> + return btrfs_csums[t].size;
> }
>
> static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
> diff --git a/disk-io.c b/disk-io.c
> index 810c2e14294a..ce0b746f4db9 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> #include "common/device-scan.h"
> +#include "crypto/hash.h"
>
> /* specified errno for check_tree_block */
> #define BTRFS_BAD_BYTENR (-1)
> @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
> crc = crc32c(crc, data, len);
> put_unaligned_le32(~crc, out);
> return 0;
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return hash_xxhash(data, len, out);
> default:
> fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
> ASSERT(0);
> @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
> }
>
> csum_type = btrfs_super_csum_type(sb);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
> error("unsupported checksum algorithm %u", csum_type);
> return -EIO;
> }
> - csum_size = btrfs_csum_sizes[csum_type];
> + csum_size = btrfs_csums[csum_type].size;
>
> btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
> result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> diff --git a/image/main.c b/image/main.c
> index 0c8ffede56f5..1265152cf524 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
>
> static void csum_block(u8 *buf, size_t len)
> {
> - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
> + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
> + u8 result[csum_size];
> u32 crc = ~(u32)0;
> crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> put_unaligned_le32(~crc, result);
> - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
> + memcpy(buf, result, csum_size);
> }
>
> static int has_name(struct btrfs_key *key)
> diff --git a/mkfs/common.c b/mkfs/common.c
> index 4a417bd7a306..939be5eb2dc2 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* write back root tree */
> @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
> btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
> if (ret != cfg->nodesize) {
> @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
> if (ret != cfg->nodesize) {
> @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
> btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
> if (ret != cfg->nodesize) {
> @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
> btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
> if (ret != cfg->nodesize) {
> @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
> if (ret != cfg->nodesize) {
> @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
> memcpy(buf->data, &super, sizeof(super));
> buf->len = BTRFS_SUPER_INFO_SIZE;
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> cfg->blocks[MKFS_SUPER_BLOCK]);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index e96cbc5399a2..64806dac7706 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> {
> if (strcasecmp(s, "crc32c") == 0) {
> return BTRFS_CSUM_TYPE_CRC32;
> + } else if (strcasecmp(s, "xxhash64") == 0 ||
> + strcasecmp(s, "xxhash") == 0) {
> + return BTRFS_CSUM_TYPE_XXHASH;
> } else {
> error("unknown csum type %s", s);
> exit(1);
> @@ -1376,7 +1379,8 @@ raid_groups:
> pretty_size(allocation.system));
> printf("SSD detected: %s\n", ssd ? "yes" : "no");
> btrfs_parse_features_to_string(features_buf, features);
> - printf("Incompat features: %s", features_buf);
> + printf("Incompat features: %s\n", features_buf);
> + printf("Checksum: %s", btrfs_csums[csum_type].name);
> printf("\n");
>
> list_all_devices(root);
>
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
2019-09-04 8:31 ` Nikolay Borisov
@ 2019-09-24 14:23 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:23 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:44PM +0200, Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
>
> Add xxhash64 as another checksumming algorithm.
>
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>
> ---
> Changes to v3:
> - Fix usage of is_valid_csum_type() (Nikolay)
> - Remove unrelated whitespace change (Nikolay)
>
> Changes to v2:
> - Integrated comments from Nikolay
> ---
> Makefile | 3 ++-
> cmds/inspect-dump-super.c | 24 +++++++++++++++---------
> convert/common.c | 2 +-
> convert/main.c | 2 +-
> crypto/hash.c | 16 ++++++++++++++++
> crypto/hash.h | 10 ++++++++++
> ctree.h | 14 ++++++++++----
> disk-io.c | 7 +++++--
> image/main.c | 5 +++--
> mkfs/common.c | 14 +++++++-------
> mkfs/main.c | 6 +++++-
> 11 files changed, 75 insertions(+), 28 deletions(-)
> create mode 100644 crypto/hash.c
> create mode 100644 crypto/hash.h
>
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> kernel-lib/crc32c.o common/messages.o \
> - uuid-tree.o utils-lib.o common/rbtree-utils.o
> + uuid-tree.o utils-lib.o common/rbtree-utils.o \
> + crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 58bf82b0bbd3..f9f38751f429 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
> super_flags_num, BTRFS_SUPER_FLAG_SUPP);
> }
>
> +static bool is_valid_csum_type(u16 csum_type)
Please split generic infrastructure updates to a separate patch, and add
the next hash on top of that.
> +{
> + switch (csum_type) {
> + case BTRFS_CSUM_TYPE_CRC32:
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> static void dump_superblock(struct btrfs_super_block *sb, int full)
> {
> int i;
> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> csum_type = btrfs_super_csum_type(sb);
> csum_size = BTRFS_CSUM_SIZE;
> printf("csum_type\t\t%hu (", csum_type);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (!is_valid_csum_type(csum_type)) {
> printf("INVALID");
> } else {
> - if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
> - printf("crc32c");
> - csum_size = btrfs_csum_sizes[csum_type];
> - } else {
> - printf("unknown");
> - }
> + printf("%s", btrfs_csums[csum_type].name);
> + csum_size = btrfs_csums[csum_type].size;
> }
> printf(")\n");
> printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
> @@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> printf("csum\t\t\t0x");
> for (i = 0, p = sb->csum; i < csum_size; i++)
> printf("%02x", p[i]);
> - if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> + if (!is_valid_csum_type(csum_type))
> printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> else if (check_csum_sblock(sb, csum_size, csum_type))
> printf(" [match]");
> diff --git a/convert/common.c b/convert/common.c
> index 2e2318a5863e..5dd1a2644bf6 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
> {
> int ret;
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* Temporary extent buffer is always mapped 1:1 on disk */
> diff --git a/convert/main.c b/convert/main.c
> index 5e6b12431f59..5eb2a59fb68a 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
> BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
> btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
> btrfs_super_csum_type(super));
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> BTRFS_SUPER_INFO_OFFSET);
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..fda7fc4e9f23
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> + XXH64_hash_t hash;
> +
> + hash = XXH64(buf, length, 0);
> + /* NOTE: we're not taking the canonical form here but the plain hash to
> + * be compatible with the kernel implementation!
> + */
/*
* Comment
*/
> + memcpy(out, &hash, 8);
> +
> + return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX 32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.h b/ctree.h
> index 870d9f4948de..4ded8161d149 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
> /* csum types */
> enum btrfs_csum_type {
> BTRFS_CSUM_TYPE_CRC32 = 0,
> + BTRFS_CSUM_TYPE_XXHASH = 1,
> };
>
> -/* four bytes for CRC32 */
> -static int btrfs_csum_sizes[] = { 4 };
> +static struct btrfs_csum {
> + u16 size;
> + const char *name;
> +} btrfs_csums[] = {
> + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
> +};
This should probably go to .c, same as in kernel so we don't have the
defintion copied in all object files that inclde ctree.h.
> #define BTRFS_EMPTY_DIR_SIZE 0
>
> @@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
> static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
> {
> int t = btrfs_super_csum_type(s);
> - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
> - return btrfs_csum_sizes[t];
> + BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
> + return btrfs_csums[t].size;
> }
>
> static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
> diff --git a/disk-io.c b/disk-io.c
> index 810c2e14294a..ce0b746f4db9 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> #include "common/device-scan.h"
> +#include "crypto/hash.h"
>
> /* specified errno for check_tree_block */
> #define BTRFS_BAD_BYTENR (-1)
> @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
> crc = crc32c(crc, data, len);
> put_unaligned_le32(~crc, out);
> return 0;
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return hash_xxhash(data, len, out);
> default:
> fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
> ASSERT(0);
> @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
> }
>
> csum_type = btrfs_super_csum_type(sb);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
> error("unsupported checksum algorithm %u", csum_type);
> return -EIO;
> }
> - csum_size = btrfs_csum_sizes[csum_type];
> + csum_size = btrfs_csums[csum_type].size;
>
> btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
> result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> diff --git a/image/main.c b/image/main.c
> index 0c8ffede56f5..1265152cf524 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
>
> static void csum_block(u8 *buf, size_t len)
> {
> - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
> + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
> + u8 result[csum_size];
> u32 crc = ~(u32)0;
> crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> put_unaligned_le32(~crc, result);
> - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
> + memcpy(buf, result, csum_size);
> }
>
> static int has_name(struct btrfs_key *key)
> diff --git a/mkfs/common.c b/mkfs/common.c
> index 4a417bd7a306..939be5eb2dc2 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* write back root tree */
> @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
> btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
> if (ret != cfg->nodesize) {
> @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
> if (ret != cfg->nodesize) {
> @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
> btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
> if (ret != cfg->nodesize) {
> @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
> btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
> if (ret != cfg->nodesize) {
> @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
> if (ret != cfg->nodesize) {
> @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
> memcpy(buf->data, &super, sizeof(super));
> buf->len = BTRFS_SUPER_INFO_SIZE;
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> cfg->blocks[MKFS_SUPER_BLOCK]);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index e96cbc5399a2..64806dac7706 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> {
> if (strcasecmp(s, "crc32c") == 0) {
> return BTRFS_CSUM_TYPE_CRC32;
> + } else if (strcasecmp(s, "xxhash64") == 0 ||
> + strcasecmp(s, "xxhash") == 0) {
> + return BTRFS_CSUM_TYPE_XXHASH;
> } else {
> error("unknown csum type %s", s);
> exit(1);
> @@ -1376,7 +1379,8 @@ raid_groups:
> pretty_size(allocation.system));
> printf("SSD detected: %s\n", ssd ? "yes" : "no");
> btrfs_parse_features_to_string(features_buf, features);
> - printf("Incompat features: %s", features_buf);
> + printf("Incompat features: %s\n", features_buf);
> + printf("Checksum: %s", btrfs_csums[csum_type].name);
> printf("\n");
>
> list_all_devices(root);
> --
> 2.16.4
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (9 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-04 8:32 ` Nikolay Borisov
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
2019-09-24 14:34 ` [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums David Sterba
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
With the introduction of xxhash64 to btrfs-progs we created a crypto/
directory for all the hashes used in btrfs (although no
cryptographically secure hash is there yet).
Move the crc32c implementation from kernel-lib/ to crypto/ as well so we
have all hashes consolidated.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Android.mk | 4 ++--
Makefile | 4 ++--
btrfs-crc.c | 2 +-
btrfs-find-root.c | 2 +-
btrfs-sb-mod.c | 2 +-
btrfs.c | 2 +-
cmds/inspect-dump-super.c | 2 +-
cmds/rescue-chunk-recover.c | 2 +-
cmds/rescue-super-recover.c | 2 +-
common/utils.c | 2 +-
convert/main.c | 2 +-
{kernel-lib => crypto}/crc32c.c | 2 +-
{kernel-lib => crypto}/crc32c.h | 0
disk-io.c | 2 +-
extent-tree.c | 2 +-
file-item.c | 2 +-
free-space-cache.c | 2 +-
hash.h | 2 +-
image/main.c | 2 +-
image/sanitize.c | 2 +-
library-test.c | 2 +-
mkfs/main.c | 2 +-
send-stream.c | 2 +-
23 files changed, 24 insertions(+), 24 deletions(-)
rename {kernel-lib => crypto}/crc32c.c (99%)
rename {kernel-lib => crypto}/crc32c.h (100%)
diff --git a/Android.mk b/Android.mk
index e8de47eb4617..8288ba7356f4 100644
--- a/Android.mk
+++ b/Android.mk
@@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \
cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \
mkfs/common.c
libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \
- kernel-lib/crc32c.c messages.c \
+ crypto/crc32c.c messages.c \
uuid-tree.c utils-lib.c rbtree-utils.c
libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
- kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+ crypto/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
blkid_objects := partition/ superblocks/ topology/
diff --git a/Makefile b/Makefile
index 45530749e2b9..c241c22d1018 100644
--- a/Makefile
+++ b/Makefile
@@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
- kernel-lib/crc32c.o common/messages.o \
+ crypto/crc32c.o common/messages.o \
uuid-tree.o utils-lib.o common/rbtree-utils.o \
crypto/hash.o crypto/xxhash.o
libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
- kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+ crypto/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h)
diff --git a/btrfs-crc.c b/btrfs-crc.c
index bcf25df8b46a..c4f81fc65f67 100644
--- a/btrfs-crc.c
+++ b/btrfs-crc.c
@@ -19,7 +19,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
void print_usage(int status)
diff --git a/btrfs-find-root.c b/btrfs-find-root.c
index e46fa8723b33..741eb9a95ac5 100644
--- a/btrfs-find-root.c
+++ b/btrfs-find-root.c
@@ -32,7 +32,7 @@
#include "kernel-lib/list.h"
#include "volumes.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "extent-cache.h"
#include "find-root.h"
#include "common/help.h"
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 105b556b0cf1..348991b39451 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -24,7 +24,7 @@
#include <string.h>
#include <limits.h>
#include <byteswap.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "disk-io.h"
#define BLOCKSIZE (4096)
diff --git a/btrfs.c b/btrfs.c
index 6c8aabe24dc8..72dad6fb3983 100644
--- a/btrfs.c
+++ b/btrfs.c
@@ -20,7 +20,7 @@
#include <getopt.h>
#include "volumes.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "cmds/commands.h"
#include "common/utils.h"
#include "common/help.h"
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index f9f38751f429..ef3d9094e661 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -32,7 +32,7 @@
#include "kernel-lib/list.h"
#include "common/utils.h"
#include "cmds/commands.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/help.h"
static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 329a608dfc6b..bf35693ddbfa 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -36,7 +36,7 @@
#include "disk-io.h"
#include "volumes.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "check/common.h"
#include "cmds/commands.h"
diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c
index ea3a00caf56c..5d6bea836c8b 100644
--- a/cmds/rescue-super-recover.c
+++ b/cmds/rescue-super-recover.c
@@ -31,7 +31,7 @@
#include "disk-io.h"
#include "kernel-lib/list.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "volumes.h"
#include "cmds/commands.h"
#include "cmds/rescue.h"
diff --git a/common/utils.c b/common/utils.c
index f2a10cccca86..fa49c01ad102 100644
--- a/common/utils.c
+++ b/common/utils.c
@@ -47,7 +47,7 @@
#include "ctree.h"
#include "disk-io.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "common/path-utils.h"
#include "common/device-scan.h"
diff --git a/convert/main.c b/convert/main.c
index 5eb2a59fb68a..fdbddc846e39 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -102,7 +102,7 @@
#include "mkfs/common.h"
#include "convert/common.h"
#include "convert/source-fs.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/fsfeatures.h"
#include "common/box.h"
diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c
similarity index 99%
rename from kernel-lib/crc32c.c
rename to crypto/crc32c.c
index 36bb6f189971..bd6283d5baeb 100644
--- a/kernel-lib/crc32c.c
+++ b/crypto/crc32c.c
@@ -8,7 +8,7 @@
*
*/
#include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include <inttypes.h>
#include <string.h>
#include <unistd.h>
diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h
similarity index 100%
rename from kernel-lib/crc32c.h
rename to crypto/crc32c.h
diff --git a/disk-io.c b/disk-io.c
index ce0b746f4db9..4093982cf3dc 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -29,7 +29,7 @@
#include "disk-io.h"
#include "volumes.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "print-tree.h"
#include "common/rbtree-utils.h"
diff --git a/extent-tree.c b/extent-tree.c
index 932af2c644bd..a8f57776bd73 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -26,7 +26,7 @@
#include "disk-io.h"
#include "print-tree.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "volumes.h"
#include "free-space-cache.h"
#include "free-space-tree.h"
diff --git a/file-item.c b/file-item.c
index c6e9d212bcab..64af57693baf 100644
--- a/file-item.c
+++ b/file-item.c
@@ -24,7 +24,7 @@
#include "disk-io.h"
#include "transaction.h"
#include "print-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/internal.h"
#define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
diff --git a/free-space-cache.c b/free-space-cache.c
index 8a57f86dc650..6e7d7e1ef561 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -22,7 +22,7 @@
#include "transaction.h"
#include "disk-io.h"
#include "extent_io.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "kernel-lib/bitops.h"
#include "common/internal.h"
#include "common/utils.h"
diff --git a/hash.h b/hash.h
index 5398e8869316..51e842047093 100644
--- a/hash.h
+++ b/hash.h
@@ -19,7 +19,7 @@
#ifndef __BTRFS_HASH_H__
#define __BTRFS_HASH_H__
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
static inline u64 btrfs_name_hash(const char *name, int len)
{
diff --git a/image/main.c b/image/main.c
index 1265152cf524..92ee28d7fe1a 100644
--- a/image/main.c
+++ b/image/main.c
@@ -28,7 +28,7 @@
#include <getopt.h>
#include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "ctree.h"
#include "disk-io.h"
#include "transaction.h"
diff --git a/image/sanitize.c b/image/sanitize.c
index 9caa5deaf2dd..cd2bd6fe2379 100644
--- a/image/sanitize.c
+++ b/image/sanitize.c
@@ -18,7 +18,7 @@
#include "common/internal.h"
#include "common/messages.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "image/sanitize.h"
#include "extent_io.h"
diff --git a/library-test.c b/library-test.c
index c44bad228e50..e47917c25830 100644
--- a/library-test.c
+++ b/library-test.c
@@ -21,7 +21,7 @@
#include "version.h"
#include "kernel-lib/rbtree.h"
#include "kernel-lib/radix-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "kernel-lib/list.h"
#include "kernel-lib/sizes.h"
#include "ctree.h"
diff --git a/mkfs/main.c b/mkfs/main.c
index 64806dac7706..a46205e7237f 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -45,7 +45,7 @@
#include "common/rbtree-utils.h"
#include "mkfs/common.h"
#include "mkfs/rootdir.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/fsfeatures.h"
#include "common/box.h"
diff --git a/send-stream.c b/send-stream.c
index 08687e5e6904..3b8ada2110aa 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -21,7 +21,7 @@
#include "send.h"
#include "send-stream.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
struct btrfs_send_stream {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
@ 2019-09-04 8:32 ` Nikolay Borisov
0 siblings, 0 replies; 23+ messages in thread
From: Nikolay Borisov @ 2019-09-04 8:32 UTC (permalink / raw)
To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist
On 3.09.19 г. 18:00 ч., Johannes Thumshirn wrote:
> With the introduction of xxhash64 to btrfs-progs we created a crypto/
> directory for all the hashes used in btrfs (although no
> cryptographically secure hash is there yet).
>
> Move the crc32c implementation from kernel-lib/ to crypto/ as well so we
> have all hashes consolidated.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> Android.mk | 4 ++--
> Makefile | 4 ++--
> btrfs-crc.c | 2 +-
> btrfs-find-root.c | 2 +-
> btrfs-sb-mod.c | 2 +-
> btrfs.c | 2 +-
> cmds/inspect-dump-super.c | 2 +-
> cmds/rescue-chunk-recover.c | 2 +-
> cmds/rescue-super-recover.c | 2 +-
> common/utils.c | 2 +-
> convert/main.c | 2 +-
> {kernel-lib => crypto}/crc32c.c | 2 +-
> {kernel-lib => crypto}/crc32c.h | 0
> disk-io.c | 2 +-
> extent-tree.c | 2 +-
> file-item.c | 2 +-
> free-space-cache.c | 2 +-
> hash.h | 2 +-
> image/main.c | 2 +-
> image/sanitize.c | 2 +-
> library-test.c | 2 +-
> mkfs/main.c | 2 +-
> send-stream.c | 2 +-
> 23 files changed, 24 insertions(+), 24 deletions(-)
> rename {kernel-lib => crypto}/crc32c.c (99%)
> rename {kernel-lib => crypto}/crc32c.h (100%)
>
> diff --git a/Android.mk b/Android.mk
> index e8de47eb4617..8288ba7356f4 100644
> --- a/Android.mk
> +++ b/Android.mk
> @@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \
> cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \
> mkfs/common.c
> libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \
> - kernel-lib/crc32c.c messages.c \
> + crypto/crc32c.c messages.c \
> uuid-tree.c utils-lib.c rbtree-utils.c
> libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> + crypto/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
> blkid_objects := partition/ superblocks/ topology/
> diff --git a/Makefile b/Makefile
> index 45530749e2b9..c241c22d1018 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> - kernel-lib/crc32c.o common/messages.o \
> + crypto/crc32c.o common/messages.o \
> uuid-tree.o utils-lib.o common/rbtree-utils.o \
> crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> + crypto/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
> libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h)
> diff --git a/btrfs-crc.c b/btrfs-crc.c
> index bcf25df8b46a..c4f81fc65f67 100644
> --- a/btrfs-crc.c
> +++ b/btrfs-crc.c
> @@ -19,7 +19,7 @@
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
>
> void print_usage(int status)
> diff --git a/btrfs-find-root.c b/btrfs-find-root.c
> index e46fa8723b33..741eb9a95ac5 100644
> --- a/btrfs-find-root.c
> +++ b/btrfs-find-root.c
> @@ -32,7 +32,7 @@
> #include "kernel-lib/list.h"
> #include "volumes.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "extent-cache.h"
> #include "find-root.h"
> #include "common/help.h"
> diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
> index 105b556b0cf1..348991b39451 100644
> --- a/btrfs-sb-mod.c
> +++ b/btrfs-sb-mod.c
> @@ -24,7 +24,7 @@
> #include <string.h>
> #include <limits.h>
> #include <byteswap.h>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "disk-io.h"
>
> #define BLOCKSIZE (4096)
> diff --git a/btrfs.c b/btrfs.c
> index 6c8aabe24dc8..72dad6fb3983 100644
> --- a/btrfs.c
> +++ b/btrfs.c
> @@ -20,7 +20,7 @@
> #include <getopt.h>
>
> #include "volumes.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "cmds/commands.h"
> #include "common/utils.h"
> #include "common/help.h"
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index f9f38751f429..ef3d9094e661 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -32,7 +32,7 @@
> #include "kernel-lib/list.h"
> #include "common/utils.h"
> #include "cmds/commands.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/help.h"
>
> static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 329a608dfc6b..bf35693ddbfa 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -36,7 +36,7 @@
> #include "disk-io.h"
> #include "volumes.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "check/common.h"
> #include "cmds/commands.h"
> diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c
> index ea3a00caf56c..5d6bea836c8b 100644
> --- a/cmds/rescue-super-recover.c
> +++ b/cmds/rescue-super-recover.c
> @@ -31,7 +31,7 @@
> #include "disk-io.h"
> #include "kernel-lib/list.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "volumes.h"
> #include "cmds/commands.h"
> #include "cmds/rescue.h"
> diff --git a/common/utils.c b/common/utils.c
> index f2a10cccca86..fa49c01ad102 100644
> --- a/common/utils.c
> +++ b/common/utils.c
> @@ -47,7 +47,7 @@
> #include "ctree.h"
> #include "disk-io.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "common/path-utils.h"
> #include "common/device-scan.h"
> diff --git a/convert/main.c b/convert/main.c
> index 5eb2a59fb68a..fdbddc846e39 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -102,7 +102,7 @@
> #include "mkfs/common.h"
> #include "convert/common.h"
> #include "convert/source-fs.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/fsfeatures.h"
> #include "common/box.h"
>
> diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c
> similarity index 99%
> rename from kernel-lib/crc32c.c
> rename to crypto/crc32c.c
> index 36bb6f189971..bd6283d5baeb 100644
> --- a/kernel-lib/crc32c.c
> +++ b/crypto/crc32c.c
> @@ -8,7 +8,7 @@
> *
> */
> #include "kerncompat.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include <inttypes.h>
> #include <string.h>
> #include <unistd.h>
> diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h
> similarity index 100%
> rename from kernel-lib/crc32c.h
> rename to crypto/crc32c.h
> diff --git a/disk-io.c b/disk-io.c
> index ce0b746f4db9..4093982cf3dc 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -29,7 +29,7 @@
> #include "disk-io.h"
> #include "volumes.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> diff --git a/extent-tree.c b/extent-tree.c
> index 932af2c644bd..a8f57776bd73 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -26,7 +26,7 @@
> #include "disk-io.h"
> #include "print-tree.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "volumes.h"
> #include "free-space-cache.h"
> #include "free-space-tree.h"
> diff --git a/file-item.c b/file-item.c
> index c6e9d212bcab..64af57693baf 100644
> --- a/file-item.c
> +++ b/file-item.c
> @@ -24,7 +24,7 @@
> #include "disk-io.h"
> #include "transaction.h"
> #include "print-tree.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/internal.h"
>
> #define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
> diff --git a/free-space-cache.c b/free-space-cache.c
> index 8a57f86dc650..6e7d7e1ef561 100644
> --- a/free-space-cache.c
> +++ b/free-space-cache.c
> @@ -22,7 +22,7 @@
> #include "transaction.h"
> #include "disk-io.h"
> #include "extent_io.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "kernel-lib/bitops.h"
> #include "common/internal.h"
> #include "common/utils.h"
> diff --git a/hash.h b/hash.h
> index 5398e8869316..51e842047093 100644
> --- a/hash.h
> +++ b/hash.h
> @@ -19,7 +19,7 @@
> #ifndef __BTRFS_HASH_H__
> #define __BTRFS_HASH_H__
>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
>
> static inline u64 btrfs_name_hash(const char *name, int len)
> {
> diff --git a/image/main.c b/image/main.c
> index 1265152cf524..92ee28d7fe1a 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -28,7 +28,7 @@
> #include <getopt.h>
>
> #include "kerncompat.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "ctree.h"
> #include "disk-io.h"
> #include "transaction.h"
> diff --git a/image/sanitize.c b/image/sanitize.c
> index 9caa5deaf2dd..cd2bd6fe2379 100644
> --- a/image/sanitize.c
> +++ b/image/sanitize.c
> @@ -18,7 +18,7 @@
> #include "common/internal.h"
> #include "common/messages.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "image/sanitize.h"
> #include "extent_io.h"
>
> diff --git a/library-test.c b/library-test.c
> index c44bad228e50..e47917c25830 100644
> --- a/library-test.c
> +++ b/library-test.c
> @@ -21,7 +21,7 @@
> #include "version.h"
> #include "kernel-lib/rbtree.h"
> #include "kernel-lib/radix-tree.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "kernel-lib/list.h"
> #include "kernel-lib/sizes.h"
> #include "ctree.h"
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 64806dac7706..a46205e7237f 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -45,7 +45,7 @@
> #include "common/rbtree-utils.h"
> #include "mkfs/common.h"
> #include "mkfs/rootdir.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/fsfeatures.h"
> #include "common/box.h"
>
> diff --git a/send-stream.c b/send-stream.c
> index 08687e5e6904..3b8ada2110aa 100644
> --- a/send-stream.c
> +++ b/send-stream.c
> @@ -21,7 +21,7 @@
>
> #include "send.h"
> #include "send-stream.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
>
> struct btrfs_send_stream {
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (10 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-24 14:34 ` [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums David Sterba
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add test-cases for creating a file-system xxhash64 as checksumming
algorithm.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 6e295274119d..3fa3c8ad42d1 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -46,6 +46,8 @@ test_mkfs_single -d single -m dup
test_mkfs_single -d dup -m single
test_mkfs_single -d dup -m dup
test_mkfs_single -d dup -m dup --mixed
+test_mkfs_single -C xxhash64
+test_mkfs_single -C xxhash
test_mkfs_multi
test_mkfs_multi -d single -m single
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
@ 2019-09-24 14:26 ` David Sterba
0 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:26 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:46PM +0200, Johannes Thumshirn wrote:
> Add test-cases for creating a file-system xxhash64 as checksumming
> algorithm.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
> index 6e295274119d..3fa3c8ad42d1 100755
> --- a/tests/mkfs-tests/001-basic-profiles/test.sh
> +++ b/tests/mkfs-tests/001-basic-profiles/test.sh
> @@ -46,6 +46,8 @@ test_mkfs_single -d single -m dup
> test_mkfs_single -d dup -m single
> test_mkfs_single -d dup -m dup
> test_mkfs_single -d dup -m dup --mixed
> +test_mkfs_single -C xxhash64
> +test_mkfs_single -C xxhash
We'll want to do full test coverage with any of the checksum selected
externally, ie. ina similar way how the 'btrfs check --mode=lowmem' is
done
$ make TEST_ENABLE_OVERRIDE=true TEST_ARGS_CHECK=--mode=lowmem test-check
The support code in the testing sicripts is not there but can be copied
and adapted for mkfs.
A separate test that quickly enumerates over the supported mkfs hashes
would still make sense.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (11 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
@ 2019-09-24 14:34 ` David Sterba
12 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:34 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:34PM +0200, Johannes Thumshirn wrote:
> Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
> prepared for an easy addition of new checksums, this patchset implements
> XXHASH64 as a second, fast but not cryptographically secure checksum hash.
>
> For changes since v2, please see the individual patches. Additionally a patch
> moving the CRC32C implementation from kernel-lib/ to crypto/ was added.
>
> For changes since v1, please see the individual patches. Additionally a unit
> test was added for regression testing this series.
>
>
> David Sterba (3):
> btrfs-progs: update checksumming api
> btrfs-progs: add xxhash sources
> btrfs-progs: add xxhash64 as checksum algorithm
>
> Johannes Thumshirn (9):
> btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
> btrfs-progs: cache csum_type in recover_control
> btrfs-progs: add checksum type to checksumming functions
> btrfs-progs: don't assume checksums are always 4 bytes
> btrfs-progs: pass checksum type to
> btrfs_csum_data()/btrfs_csum_final()
> btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
> btrfs-progs: add option for checksum type to mkfs
> btrfs-progs: move crc32c implementation to crypto/
> btrfs-progs: add test-case for mkfs with xxhash64
1-9 now in devel, please split 10 and see the comments for 12. You can
also update the documentation, I think the user interface is not going
to change significantly.
For first release we can use the builtin xxhash but as there's a
standalone library we'll need to add support for configure-time
selection. This will be even more necessary once SHA256 is added.
^ permalink raw reply [flat|nested] 23+ messages in thread