* [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums
@ 2019-09-03 15:00 Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
` (12 more replies)
0 siblings, 13 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
prepared for an easy addition of new checksums, this patchset implements
XXHASH64 as a second, fast but not cryptographically secure checksum hash.
For changes since v2, please see the individual patches. Additionally a patch
moving the CRC32C implementation from kernel-lib/ to crypto/ was added.
For changes since v1, please see the individual patches. Additionally a unit
test was added for regression testing this series.
David Sterba (3):
btrfs-progs: update checksumming api
btrfs-progs: add xxhash sources
btrfs-progs: add xxhash64 as checksum algorithm
Johannes Thumshirn (9):
btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
btrfs-progs: cache csum_type in recover_control
btrfs-progs: add checksum type to checksumming functions
btrfs-progs: don't assume checksums are always 4 bytes
btrfs-progs: pass checksum type to
btrfs_csum_data()/btrfs_csum_final()
btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
btrfs-progs: add option for checksum type to mkfs
btrfs-progs: move crc32c implementation to crypto/
btrfs-progs: add test-case for mkfs with xxhash64
Android.mk | 4 +-
Makefile | 7 +-
btrfs-corrupt-block.c | 3 +-
btrfs-crc.c | 2 +-
btrfs-find-root.c | 2 +-
btrfs-sb-mod.c | 32 +-
btrfs.c | 2 +-
check/main.c | 20 +-
cmds/inspect-dump-super.c | 38 +-
cmds/rescue-chunk-recover.c | 25 +-
cmds/rescue-super-recover.c | 2 +-
common/utils.c | 2 +-
convert/common.c | 14 +-
convert/main.c | 5 +-
{kernel-lib => crypto}/crc32c.c | 2 +-
{kernel-lib => crypto}/crc32c.h | 0
crypto/hash.c | 16 +
crypto/hash.h | 10 +
crypto/xxhash.c | 1025 +++++++++++++++++++++++++++
crypto/xxhash.h | 445 ++++++++++++
ctree.h | 18 +-
disk-io.c | 82 ++-
disk-io.h | 8 +-
extent-tree.c | 2 +-
file-item.c | 13 +-
free-space-cache.c | 4 +-
hash.h | 2 +-
image/main.c | 9 +-
image/sanitize.c | 2 +-
library-test.c | 2 +-
mkfs/common.c | 23 +-
mkfs/common.h | 2 +
mkfs/main.c | 29 +-
send-stream.c | 2 +-
tests/mkfs-tests/001-basic-profiles/test.sh | 2 +
35 files changed, 1710 insertions(+), 146 deletions(-)
rename {kernel-lib => crypto}/crc32c.c (99%)
rename {kernel-lib => crypto}/crc32c.h (100%)
create mode 100644 crypto/hash.c
create mode 100644 crypto/hash.h
create mode 100644 crypto/xxhash.c
create mode 100644 crypto/xxhash.h
--
2.16.4
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 12:56 ` David Sterba
2019-09-24 13:06 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Johannes Thumshirn
` (11 subsequent siblings)
12 siblings, 2 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
The callers of csum_tree_block_size() blindly assume we're only having
crc32c as a possible checksum and thus pass in
btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
csum_tree_block_size().
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
mkfs/common.c | 14 +++++++-------
mkfs/common.h | 2 ++
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/mkfs/common.c b/mkfs/common.c
index caca5e707233..b6e549b19272 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
/* write back root tree */
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
if (ret != BTRFS_SUPER_INFO_SIZE) {
diff --git a/mkfs/common.h b/mkfs/common.h
index 28912906d0a9..1ca71a4fcce5 100644
--- a/mkfs/common.h
+++ b/mkfs/common.h
@@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
u64 features;
/* Size of the filesystem in bytes */
u64 num_bytes;
+ /* checksum algorithm to use */
+ enum btrfs_csum_type csum_type;
/* Output fields, set during creation */
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Johannes Thumshirn
` (10 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Cache the super-block's checksum type field in 'struct recover_control'.
This will be needed for further refactoring the checksum handling.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
cmds/rescue-chunk-recover.c | 2 ++
ctree.h | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index c9b268781159..1959a2047c17 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -47,6 +47,7 @@ struct recover_control {
int yes;
u16 csum_size;
+ u16 csum_type;
u32 sectorsize;
u32 nodesize;
u64 generation;
@@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path)
rc->generation = btrfs_super_generation(sb);
rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb);
rc->csum_size = btrfs_super_csum_size(sb);
+ rc->csum_type = btrfs_super_csum_type(sb);
/* if seed, the result of scanning below will be partial */
if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) {
diff --git a/ctree.h b/ctree.h
index 0d12563b7261..870d9f4948de 100644
--- a/ctree.h
+++ b/ctree.h
@@ -165,7 +165,9 @@ struct btrfs_free_space_ctl;
#define BTRFS_CSUM_SIZE 32
/* csum types */
-#define BTRFS_CSUM_TYPE_CRC32 0
+enum btrfs_csum_type {
+ BTRFS_CSUM_TYPE_CRC32 = 0,
+};
/* four bytes for CRC32 */
static int btrfs_csum_sizes[] = { 4 };
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Johannes Thumshirn
` (9 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
and verify_tree_block_csum_silent().
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-corrupt-block.c | 3 ++-
cmds/rescue-chunk-recover.c | 3 ++-
convert/common.c | 3 ++-
convert/main.c | 3 ++-
disk-io.c | 21 ++++++++++++---------
disk-io.h | 5 +++--
mkfs/common.c | 21 ++++++++++++++-------
7 files changed, 37 insertions(+), 22 deletions(-)
diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index bbef0c02e5d1..1dde9594bdcc 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
if (!trans) {
u16 csum_size =
btrfs_super_csum_size(fs_info->super_copy);
- csum_tree_block_size(eb, csum_size, 0);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+ csum_tree_block_size(eb, csum_size, 0, csum_type);
write_extent_to_disk(eb);
}
}
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 1959a2047c17..c2a26ad040ee 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
continue;
}
- if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
+ if (verify_tree_block_csum_silent(buf, rc->csum_size,
+ rc->csum_type)) {
bytenr += rc->sectorsize;
continue;
}
diff --git a/convert/common.c b/convert/common.c
index dea5f5b20d50..f8bbb23cba89 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
{
int ret;
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
/* Temporary extent buffer is always mapped 1:1 on disk */
ret = pwrite(fd, buf->data, buf->len, bytenr);
diff --git a/convert/main.c b/convert/main.c
index 9711874bd137..5e6b12431f59 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+ btrfs_super_csum_type(super));
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
BTRFS_SUPER_INFO_OFFSET);
if (ret != BTRFS_SUPER_INFO_SIZE)
diff --git a/disk-io.c b/disk-io.c
index 01314504a50a..a4995a628210 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
- int verify, int silent)
+ int verify, int silent, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
u32 len;
@@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
return 0;
}
-int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
+int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
+ u16 csum_type)
{
- return __csum_tree_block_size(buf, csum_size, verify, 0);
+ return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
}
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+ u16 csum_type)
{
- return __csum_tree_block_size(buf, csum_size, 1, 1);
+ return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
}
int csum_tree_block(struct btrfs_fs_info *fs_info,
struct extent_buffer *buf, int verify)
{
- u16 csum_size =
- btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
if (verify && fs_info->suppress_check_block_errors)
- return verify_tree_block_csum_silent(buf, csum_size);
- return csum_tree_block_size(buf, csum_size, verify);
+ return verify_tree_block_csum_silent(buf, csum_size, csum_type);
+ return csum_tree_block_size(buf, csum_size, verify, csum_type);
}
struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
diff --git a/disk-io.h b/disk-io.h
index 7b5c3806ba98..394997ad72cb 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
- int verify);
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
+ int verify, u16 csum_type);
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+ u16 csum_type);
int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
int write_tree_block(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info,
diff --git a/mkfs/common.c b/mkfs/common.c
index b6e549b19272..9762391a8d2b 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
/* write back root tree */
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+ csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ cfg->csum_type);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
if (ret != BTRFS_SUPER_INFO_SIZE) {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (2 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Johannes Thumshirn
` (8 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-sb-mod.c | 9 ++++++---
check/main.c | 2 +-
cmds/inspect-dump-super.c | 3 ++-
cmds/rescue-chunk-recover.c | 2 +-
convert/common.c | 3 ++-
disk-io.c | 12 ++++++------
disk-io.h | 2 +-
file-item.c | 2 +-
8 files changed, 20 insertions(+), 15 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 16a26f772494..932c2a0432ef 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb)
u32 crc = ~(u32)0;
crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
- crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
return !memcmp(sb, &result, csum_size);
@@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb)
u32 crc = ~(u32)0;
if (is_sb) {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
} else {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BLOCKSIZE - BTRFS_CSUM_SIZE);
}
btrfs_csum_final(crc, result);
diff --git a/check/main.c b/check/main.c
index 2e16b4e6f05b..0f02c85368ec 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5660,7 +5660,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
tmp = offset + data_checked;
csum = btrfs_csum_data((char *)data + tmp,
- csum, fs_info->sectorsize);
+ (u8 *)&csum, fs_info->sectorsize);
btrfs_csum_final(csum, (u8 *)&csum);
csum_offset = leaf_offset +
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 65fb3506eac6..96ad3deca3d8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size)
u32 crc = ~(u32)0;
crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
- crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
return !memcmp(sb, &result, csum_size);
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index c2a26ad040ee..37c024ed8965 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(data, csum_result, len);
+ csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
btrfs_csum_final(csum_result, (u8 *)&csum_result);
if (csum_result != tree_csum)
ret = 1;
diff --git a/convert/common.c b/convert/common.c
index f8bbb23cba89..ab8e6b9f4749 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u32 crc = ~(u32)0;
int ret;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
diff --git a/disk-io.c b/disk-io.c
index a4995a628210..a0c37c569d58 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(char *data, u32 seed, size_t len)
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
{
- return crc32c(seed, data, len);
+ return crc32c(*(u32*)seed, data, len);
}
void btrfs_csum_final(u32 crc, u8 *result)
@@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len);
+ crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
btrfs_csum_final(crc, result);
if (verify) {
@@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
csum_size = btrfs_csum_sizes[csum_type];
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, result);
@@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
@@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+ crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
btrfs_csum_final(crc, &sb->csum[0]);
diff --git a/disk-io.h b/disk-io.h
index 394997ad72cb..92c87f28f8b2 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u32 seed, size_t len);
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
void btrfs_csum_final(u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
diff --git a/file-item.c b/file-item.c
index 3bf48c68913d..5f6548e9a74f 100644
--- a/file-item.c
+++ b/file-item.c
@@ -312,7 +312,7 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(data, csum_result, len);
+ csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
btrfs_csum_final(csum_result, (u8 *)&csum_result);
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final()
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (3 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Johannes Thumshirn
` (7 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
In preparation to supporting new checksum algorithm pass the checksum type
to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate any
differences in processing into the respective functions
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Changes to v1:
- Update changelog and squashed the other identical patch
---
btrfs-sb-mod.c | 15 ++++++++++-----
check/main.c | 6 ++++--
cmds/inspect-dump-super.c | 8 ++++----
cmds/rescue-chunk-recover.c | 10 ++++++----
convert/common.c | 5 +++--
disk-io.c | 39 +++++++++++++++++++++++++++------------
disk-io.h | 4 ++--
file-item.c | 7 +++++--
free-space-cache.c | 2 +-
image/main.c | 2 +-
10 files changed, 63 insertions(+), 35 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 932c2a0432ef..d9630f187d0f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb)
{
u8 result[csum_size];
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(sb);
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
return !memcmp(sb, &result, csum_size);
}
@@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb)
u8 result[csum_size];
struct btrfs_header *hdr;
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(block);
if (is_sb) {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)block + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
} else {
- crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type,
+ (char *)block + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BLOCKSIZE - BTRFS_CSUM_SIZE);
}
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index 0f02c85368ec..1896a28f157d 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5619,6 +5619,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
struct btrfs_fs_info *fs_info = root->fs_info;
u64 offset = 0;
u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+ u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
char *data;
unsigned long csum_offset;
u32 csum;
@@ -5659,9 +5660,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
csum = ~(u32)0;
tmp = offset + data_checked;
- csum = btrfs_csum_data((char *)data + tmp,
+ csum = btrfs_csum_data(csum_type,
+ (char *)data + tmp,
(u8 *)&csum, fs_info->sectorsize);
- btrfs_csum_final(csum, (u8 *)&csum);
+ btrfs_csum_final(csum_type, csum, (u8 *)&csum);
csum_offset = leaf_offset +
tmp / fs_info->sectorsize * csum_size;
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 96ad3deca3d8..40019a6670ef 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -35,15 +35,15 @@
#include "kernel-lib/crc32c.h"
#include "common/help.h"
-static int check_csum_sblock(void *sb, int csum_size)
+static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
u32 crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
return !memcmp(sb, &result, csum_size);
}
@@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
printf(" [UNKNOWN CSUM TYPE OR SIZE]");
- else if (check_csum_sblock(sb, csum_size))
+ else if (check_csum_sblock(sb, csum_size, csum_type))
printf(" [match]");
else
printf(" [DON'T MATCH]");
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 37c024ed8965..d556323ec34a 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
return dev_offset + data_offset;
}
-static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
+static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
+ u16 csum_type)
{
char *data;
int ret = 0;
@@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_result, (u8 *)&csum_result);
+ csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+ btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
if (csum_result != tree_csum)
ret = 1;
out:
@@ -2102,7 +2103,8 @@ next_csum:
devext->objectid, 1));
ret = check_one_csum(dev->fd, data_offset, blocksize,
- tree_csum);
+ tree_csum,
+ btrfs_super_csum_type(root->fs_info->super_copy));
if (ret < 0)
goto fail_out;
else if (ret > 0)
diff --git a/convert/common.c b/convert/common.c
index ab8e6b9f4749..894a6ee0ba90 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u64 sb_bytenr)
{
u32 crc = ~(u32)0;
+ u16 csum_type = btrfs_super_csum_type(sb);
int ret;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
(u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
if (ret < BTRFS_SUPER_INFO_SIZE)
ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index a0c37c569d58..7e538969c57a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
{
- return crc32c(*(u32*)seed, data, len);
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ return crc32c(*(u32*)seed, data, len);
+ default: /* Not reached */
+ return ~(u32)0;
+ }
+
}
-void btrfs_csum_final(u32 crc, u8 *result)
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
{
- put_unaligned_le32(~crc, result);
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ put_unaligned_le32(~crc, result);
+ break;
+ default: /* Not reached */
+ break;
+ }
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
- btrfs_csum_final(crc, result);
+ crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
+ (u8 *)&crc, len);
+ btrfs_csum_final(csum_type, crc, result);
if (verify) {
if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
@@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
csum_size = btrfs_csum_sizes[csum_type];
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ btrfs_csum_final(csum_type, crc, result);
if (memcmp(result, sb->csum, csum_size)) {
error("superblock checksum mismatch");
@@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
u64 bytenr;
u32 crc;
int i, ret;
+ u16 csum_type = btrfs_super_csum_type(sb);
/*
* We need to write super block after all metadata written.
@@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
crc = ~(u32)0;
- crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, &sb->csum[0]);
+ btrfs_csum_final(csum_type, crc, &sb->csum[0]);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 92c87f28f8b2..4b5e9ea86385 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u32 crc, u8 *result);
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 5f6548e9a74f..78fdcecd0bab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
u16 csum_size =
btrfs_super_csum_size(root->fs_info->super_copy);
+ u16 csum_type =
+ btrfs_super_csum_type(root->fs_info->super_copy);
+
path = btrfs_alloc_path();
if (!path)
return -ENOMEM;
@@ -312,8 +315,8 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_result, (u8 *)&csum_result);
+ csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+ btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
(unsigned long long)bytenr);
diff --git a/free-space-cache.c b/free-space-cache.c
index e872eb6a00db..8a57f86dc650 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
io_ctl_map_page(io_ctl, 0);
crc = crc32c(crc, io_ctl->orig + offset,
io_ctl->root->fs_info->sectorsize - offset);
- btrfs_csum_final(crc, (u8 *)&crc);
+ put_unaligned_le32(~crc, (u8 *)&crc);
if (val != crc) {
printk("btrfs: csum mismatch on free space cache\n");
io_ctl_unmap_page(io_ctl);
diff --git a/image/main.c b/image/main.c
index 28ef1609b5ff..0c8ffede56f5 100644
--- a/image/main.c
+++ b/image/main.c
@@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
u32 crc = ~(u32)0;
crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
- btrfs_csum_final(crc, result);
+ put_unaligned_le32(~crc, result);
memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
}
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (4 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 07/12] btrfs-progs: update checksumming api Johannes Thumshirn
` (6 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb'
argument set to 1.
Get rid of the special case for is_sb == 0.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
btrfs-sb-mod.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index d9630f187d0f..9e64b34d2b8f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb)
return !memcmp(sb, &result, csum_size);
}
-static void update_block_csum(void *block, int is_sb)
+static void update_block_csum(void *block)
{
u8 result[csum_size];
struct btrfs_header *hdr;
u32 crc = ~(u32)0;
u16 csum_type = btrfs_super_csum_type(block);
- if (is_sb) {
- crc = btrfs_csum_data(csum_type,
- (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- } else {
- crc = btrfs_csum_data(csum_type,
- (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BLOCKSIZE - BTRFS_CSUM_SIZE);
- }
+ crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+ (u8 *)&crc,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+
btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
@@ -354,7 +347,7 @@ int main(int argc, char **argv) {
if (changed) {
printf("Update csum\n");
- update_block_csum(buf, 1);
+ update_block_csum(buf);
ret = pwrite(fd, buf, BLOCKSIZE, off);
if (ret <= 0) {
printf("pwrite error %d at offset %llu\n",
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 07/12] btrfs-progs: update checksumming api
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (5 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
` (5 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
Update the checksumming API to be able to cope with more checksum types
than just CRC32C.
Co-developed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
btrfs-sb-mod.c | 21 ++++++----------
check/main.c | 20 +++++++--------
cmds/inspect-dump-super.c | 9 +++----
cmds/rescue-chunk-recover.c | 12 ++++++---
convert/common.c | 9 +++----
disk-io.c | 59 ++++++++++++++++++---------------------------
disk-io.h | 3 +--
file-item.c | 8 +++---
8 files changed, 60 insertions(+), 81 deletions(-)
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 9e64b34d2b8f..105b556b0cf1 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -33,31 +33,24 @@ static int csum_size;
static int check_csum_superblock(void *sb)
{
- u8 result[csum_size];
- u32 crc = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
u16 csum_type = btrfs_super_csum_type(sb);
- crc = btrfs_csum_data(csum_type,
- (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- return !memcmp(sb, &result, csum_size);
+ return !memcmp(sb, result, csum_size);
}
static void update_block_csum(void *block)
{
- u8 result[csum_size];
+ u8 result[BTRFS_CSUM_SIZE];
struct btrfs_header *hdr;
- u32 crc = ~(u32)0;
u16 csum_type = btrfs_super_csum_type(block);
- crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
memset(block, 0, BTRFS_CSUM_SIZE);
hdr = (struct btrfs_header *)block;
memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index 1896a28f157d..5028599a773a 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5620,10 +5620,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
u64 offset = 0;
u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
- char *data;
+ u8 *data;
unsigned long csum_offset;
- u32 csum;
- u32 csum_expected;
+ u8 result[BTRFS_CSUM_SIZE];
+ u8 csum_expected[BTRFS_CSUM_SIZE];
u64 read_len;
u64 data_checked = 0;
u64 tmp;
@@ -5649,7 +5649,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
for (mirror = 1; mirror <= num_copies; mirror++) {
read_len = num_bytes - offset;
/* read as much space once a time */
- ret = read_extent_data(fs_info, data + offset,
+ ret = read_extent_data(fs_info, (char *)data + offset,
bytenr + offset, &read_len, mirror);
if (ret)
goto out;
@@ -5657,24 +5657,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
data_checked = 0;
/* verify every 4k data's checksum */
while (data_checked < read_len) {
- csum = ~(u32)0;
tmp = offset + data_checked;
- csum = btrfs_csum_data(csum_type,
- (char *)data + tmp,
- (u8 *)&csum, fs_info->sectorsize);
- btrfs_csum_final(csum_type, csum, (u8 *)&csum);
+ btrfs_csum_data(csum_type, data + tmp,
+ result, fs_info->sectorsize);
csum_offset = leaf_offset +
tmp / fs_info->sectorsize * csum_size;
read_extent_buffer(eb, (char *)&csum_expected,
csum_offset, csum_size);
- if (csum != csum_expected) {
+ if (memcmp(result, csum_expected, csum_size) != 0) {
csum_mismatch = true;
+ /* FIXME: format */
fprintf(stderr,
"mirror %d bytenr %llu csum %u expected csum %u\n",
mirror, bytenr + tmp,
- csum, csum_expected);
+ result[0], csum_expected[0]);
}
data_checked += fs_info->sectorsize;
}
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 40019a6670ef..58bf82b0bbd3 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -38,14 +38,11 @@
static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
{
u8 result[BTRFS_CSUM_SIZE];
- u32 crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- return !memcmp(sb, &result, csum_size);
+ return !memcmp(sb, result, csum_size);
}
static void print_sys_chunk_array(struct btrfs_super_block *sb)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index d556323ec34a..329a608dfc6b 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
{
char *data;
int ret = 0;
- u32 csum_result = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
+ int csum_size = 0;
+ u8 expected_csum[BTRFS_CSUM_SIZE];
+
+ ASSERT(0);
data = malloc(len);
if (!data)
@@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
goto out;
}
ret = 0;
- csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
- if (csum_result != tree_csum)
+ put_unaligned_le32(tree_csum, expected_csum);
+ btrfs_csum_data(csum_type, (u8 *)data, result, len);
+ if (memcmp(result, expected_csum, csum_size) != 0)
ret = 1;
out:
free(data);
diff --git a/convert/common.c b/convert/common.c
index 894a6ee0ba90..8cae507ec0f7 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
u64 sb_bytenr)
{
- u32 crc = ~(u32)0;
+ u8 result[BTRFS_CSUM_SIZE];
u16 csum_type = btrfs_super_csum_type(sb);
int ret;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
if (ret < BTRFS_SUPER_INFO_SIZE)
ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index 7e538969c57a..810c2e14294a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
}
}
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
{
- switch (csum_type) {
- case BTRFS_CSUM_TYPE_CRC32:
- return crc32c(*(u32*)seed, data, len);
- default: /* Not reached */
- return ~(u32)0;
- }
-
-}
+ u32 crc = ~(u32)0;
+ memset(out, 0, BTRFS_CSUM_SIZE);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
-{
switch (csum_type) {
case BTRFS_CSUM_TYPE_CRC32:
- put_unaligned_le32(~crc, result);
- break;
- default: /* Not reached */
- break;
+ crc = crc32c(crc, data, len);
+ put_unaligned_le32(~crc, out);
+ return 0;
+ default:
+ fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
+ ASSERT(0);
}
+
+ return -1;
}
static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
{
u8 result[BTRFS_CSUM_SIZE];
u32 len;
- u32 crc = ~(u32)0;
len = buf->len - BTRFS_CSUM_SIZE;
- crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
- (u8 *)&crc, len);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE,
+ result, len);
if (verify) {
if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
+ /* FIXME: format */
if (!silent)
printk("checksum verify failed on %llu found %08X wanted %08X\n",
(unsigned long long)buf->start,
- *((u32 *)result),
- *((u32*)(char *)buf->data));
+ result[0],
+ buf->data[0]);
return 1;
}
} else {
@@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
{
u8 result[BTRFS_CSUM_SIZE];
- u32 crc;
u16 csum_type;
int csum_size;
u8 *metadata_uuid;
@@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
}
csum_size = btrfs_csum_sizes[csum_type];
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
- (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, result);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+ result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
if (memcmp(result, sb->csum, csum_size)) {
error("superblock checksum mismatch");
@@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
struct btrfs_device *device)
{
u64 bytenr;
- u32 crc;
+ u8 result[BTRFS_CSUM_SIZE];
int i, ret;
u16 csum_type = btrfs_super_csum_type(sb);
@@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
}
if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
- BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
+ BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
btrfs_set_super_bytenr(sb, bytenr);
- crc = ~(u32)0;
- crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+ btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
- btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+ memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
/*
* super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 4b5e9ea86385..545cacda9a79 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len);
int btrfs_open_device(struct btrfs_device *dev);
int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 78fdcecd0bab..c6e9d212bcab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
struct btrfs_csum_item *item;
struct extent_buffer *leaf = NULL;
u64 csum_offset;
- u32 csum_result = ~(u32)0;
+ u8 csum_result[BTRFS_CSUM_SIZE];
u32 sectorsize = root->fs_info->sectorsize;
u32 nritems;
u32 ins_size;
@@ -315,14 +315,14 @@ csum:
item = (struct btrfs_csum_item *)((unsigned char *)item +
csum_offset * csum_size);
found:
- csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
- btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
+ btrfs_csum_data(csum_type, (u8 *)data, csum_result, len);
+ /* FIXME: does not make sense for non-crc32c */
if (csum_result == 0) {
printk("csum result is 0 for block %llu\n",
(unsigned long long)bytenr);
}
- write_extent_buffer(leaf, &csum_result, (unsigned long)item,
+ write_extent_buffer(leaf, csum_result, (unsigned long)item,
csum_size);
btrfs_mark_buffer_dirty(path->nodes[0]);
fail:
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (6 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 07/12] btrfs-progs: update checksumming api Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 09/12] btrfs-progs: add xxhash sources Johannes Thumshirn
` (4 subsequent siblings)
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add an option to mkfs to specify which checksum algorithm will be used for
the filesystem.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
convert/common.c | 2 +-
mkfs/common.c | 2 +-
mkfs/main.c | 21 ++++++++++++++++++++-
3 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/convert/common.c b/convert/common.c
index 8cae507ec0f7..2e2318a5863e 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -135,7 +135,7 @@ static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
super->__unused_leafsize = cpu_to_le32(cfg->nodesize);
btrfs_set_super_nodesize(super, cfg->nodesize);
btrfs_set_super_stripesize(super, cfg->stripesize);
- btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
+ btrfs_set_super_csum_type(super, cfg->csum_type);
btrfs_set_super_chunk_root(super, chunk_bytenr);
btrfs_set_super_cache_generation(super, -1);
btrfs_set_super_incompat_flags(super, cfg->features);
diff --git a/mkfs/common.c b/mkfs/common.c
index 9762391a8d2b..4a417bd7a306 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -202,7 +202,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
btrfs_set_super_nodesize(&super, cfg->nodesize);
btrfs_set_super_stripesize(&super, cfg->stripesize);
- btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32);
+ btrfs_set_super_csum_type(&super, cfg->csum_type);
btrfs_set_super_chunk_root_generation(&super, 1);
btrfs_set_super_cache_generation(&super, -1);
btrfs_set_super_incompat_flags(&super, cfg->features);
diff --git a/mkfs/main.c b/mkfs/main.c
index b752da13aba9..e96cbc5399a2 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -352,6 +352,7 @@ static void print_usage(int ret)
printf("\t--shrink (with --rootdir) shrink the filled filesystem to minimal size\n");
printf("\t-K|--nodiscard do not perform whole device TRIM\n");
printf("\t-f|--force force overwrite of existing filesystem\n");
+ printf("\t-C|--checksum checksum algorithm to use (default: crc32c)\n");
printf(" general:\n");
printf("\t-q|--quiet no messages except errors\n");
printf("\t-V|--version print the mkfs.btrfs version and exit\n");
@@ -386,6 +387,18 @@ static u64 parse_profile(const char *s)
return 0;
}
+static enum btrfs_csum_type parse_csum_type(const char *s)
+{
+ if (strcasecmp(s, "crc32c") == 0) {
+ return BTRFS_CSUM_TYPE_CRC32;
+ } else {
+ error("unknown csum type %s", s);
+ exit(1);
+ }
+ /* not reached */
+ return 0;
+}
+
static char *parse_label(const char *input)
{
int len = strlen(input);
@@ -832,6 +845,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
struct mkfs_allocation allocation = { 0 };
struct btrfs_mkfs_config mkfs_cfg;
+ enum btrfs_csum_type csum_type = BTRFS_CSUM_TYPE_CRC32;
crc32c_optimization_init();
@@ -841,6 +855,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
static const struct option long_options[] = {
{ "alloc-start", required_argument, NULL, 'A'},
{ "byte-count", required_argument, NULL, 'b' },
+ { "checksum", required_argument, NULL, 'C' },
{ "force", no_argument, NULL, 'f' },
{ "leafsize", required_argument, NULL, 'l' },
{ "label", required_argument, NULL, 'L'},
@@ -860,7 +875,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
{ NULL, 0, NULL, 0}
};
- c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+ c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq",
long_options, NULL);
if (c < 0)
break;
@@ -938,6 +953,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
case GETOPT_VAL_SHRINK:
shrink_rootdir = true;
break;
+ case 'C':
+ csum_type = parse_csum_type(optarg);
+ break;
case GETOPT_VAL_HELP:
default:
print_usage(c != GETOPT_VAL_HELP);
@@ -1176,6 +1194,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
mkfs_cfg.sectorsize = sectorsize;
mkfs_cfg.stripesize = stripesize;
mkfs_cfg.features = features;
+ mkfs_cfg.csum_type = csum_type;
ret = make_btrfs(fd, &mkfs_cfg);
if (ret) {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 09/12] btrfs-progs: add xxhash sources
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (7 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
` (3 subsequent siblings)
12 siblings, 0 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
git://github.com/Cyan4973/xxHash
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
crypto/xxhash.c | 1025 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
crypto/xxhash.h | 445 ++++++++++++++++++++++++
2 files changed, 1470 insertions(+)
create mode 100644 crypto/xxhash.c
create mode 100644 crypto/xxhash.h
diff --git a/crypto/xxhash.c b/crypto/xxhash.c
new file mode 100644
index 000000000000..58bb749d3078
--- /dev/null
+++ b/crypto/xxhash.c
@@ -0,0 +1,1025 @@
+/*
+* xxHash - Fast Hash algorithm
+* Copyright (C) 2012-2016, Yann Collet
+*
+* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+*
+* Redistribution and use in source and binary forms, with or without
+* modification, are permitted provided that the following conditions are
+* met:
+*
+* * Redistributions of source code must retain the above copyright
+* notice, this list of conditions and the following disclaimer.
+* * Redistributions in binary form must reproduce the above
+* copyright notice, this list of conditions and the following disclaimer
+* in the documentation and/or other materials provided with the
+* distribution.
+*
+* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*
+* You can contact the author at :
+* - xxHash homepage: http://www.xxhash.com
+* - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+
+/* *************************************
+* Tuning parameters
+***************************************/
+/*!XXH_FORCE_MEMORY_ACCESS :
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
+ * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
+ * Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
+ * It can generate buggy code on targets which do not support unaligned memory accesses.
+ * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See http://stackoverflow.com/a/32095106/646947 for details.
+ * Prefer these methods in priority order (0 > 1 > 2)
+ */
+#ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */
+# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
+ || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
+ || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+# define XXH_FORCE_MEMORY_ACCESS 2
+# elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \
+ (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
+ || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
+ || defined(__ARM_ARCH_7S__) ))
+# define XXH_FORCE_MEMORY_ACCESS 1
+# endif
+#endif
+
+/*!XXH_ACCEPT_NULL_INPUT_POINTER :
+ * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault.
+ * When this macro is enabled, xxHash actively checks input for null pointer.
+ * It it is, result for null input pointers is the same as a null-length input.
+ */
+#ifndef XXH_ACCEPT_NULL_INPUT_POINTER /* can be defined externally */
+# define XXH_ACCEPT_NULL_INPUT_POINTER 0
+#endif
+
+/*!XXH_FORCE_ALIGN_CHECK :
+ * This is a minor performance trick, only useful with lots of very small keys.
+ * It means : check for aligned/unaligned input.
+ * The check costs one initial branch per hash;
+ * set it to 0 when the input is guaranteed to be aligned,
+ * or when alignment doesn't matter for performance.
+ */
+#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
+# if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
+# define XXH_FORCE_ALIGN_CHECK 0
+# else
+# define XXH_FORCE_ALIGN_CHECK 1
+# endif
+#endif
+
+
+/* *************************************
+* Includes & Memory related functions
+***************************************/
+/*! Modify the local functions below should you wish to use some other memory routines
+* for malloc(), free() */
+#include <stdlib.h>
+static void* XXH_malloc(size_t s) { return malloc(s); }
+static void XXH_free (void* p) { free(p); }
+/*! and for memcpy() */
+#include <string.h>
+static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
+
+#include <assert.h> /* assert */
+
+#define XXH_STATIC_LINKING_ONLY
+#include "xxhash.h"
+
+
+/* *************************************
+* Compiler Specific Options
+***************************************/
+#ifdef _MSC_VER /* Visual Studio */
+# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
+# define XXH_FORCE_INLINE static __forceinline
+# define XXH_NO_INLINE static __declspec(noinline)
+#else
+# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
+# ifdef __GNUC__
+# define XXH_FORCE_INLINE static inline __attribute__((always_inline))
+# define XXH_NO_INLINE static __attribute__((noinline))
+# else
+# define XXH_FORCE_INLINE static inline
+# define XXH_NO_INLINE static
+# endif
+# else
+# define XXH_FORCE_INLINE static
+# define XXH_NO_INLINE static
+# endif /* __STDC_VERSION__ */
+#endif
+
+
+/* *************************************
+* Basic Types
+***************************************/
+#ifndef MEM_MODULE
+# if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+ typedef uint8_t BYTE;
+ typedef uint16_t U16;
+ typedef uint32_t U32;
+# else
+ typedef unsigned char BYTE;
+ typedef unsigned short U16;
+ typedef unsigned int U32;
+# endif
+#endif
+
+
+/* === Memory access === */
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; } __attribute__((packed)) unalign;
+static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+static U32 XXH_read32(const void* memPtr)
+{
+ U32 val;
+ memcpy(&val, memPtr, sizeof(val));
+ return val;
+}
+
+#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+
+/* === Endianess === */
+typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
+
+/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
+#ifndef XXH_CPU_LITTLE_ENDIAN
+static int XXH_isLittleEndian(void)
+{
+ const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */
+ return one.c[0];
+}
+# define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian()
+#endif
+
+
+
+
+/* ****************************************
+* Compiler-specific Functions and Macros
+******************************************/
+#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
+
+/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
+#if defined(_MSC_VER)
+# define XXH_rotl32(x,r) _rotl(x,r)
+# define XXH_rotl64(x,r) _rotl64(x,r)
+#else
+# define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r))))
+# define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r))))
+#endif
+
+#if defined(_MSC_VER) /* Visual Studio */
+# define XXH_swap32 _byteswap_ulong
+#elif XXH_GCC_VERSION >= 403
+# define XXH_swap32 __builtin_bswap32
+#else
+static U32 XXH_swap32 (U32 x)
+{
+ return ((x << 24) & 0xff000000 ) |
+ ((x << 8) & 0x00ff0000 ) |
+ ((x >> 8) & 0x0000ff00 ) |
+ ((x >> 24) & 0x000000ff );
+}
+#endif
+
+
+/* ***************************
+* Memory reads
+*****************************/
+typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
+
+XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
+}
+
+static U32 XXH_readBE32(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
+}
+
+XXH_FORCE_INLINE U32
+XXH_readLE32_align(const void* ptr, XXH_alignment align)
+{
+ if (align==XXH_unaligned) {
+ return XXH_readLE32(ptr);
+ } else {
+ return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
+ }
+}
+
+
+/* *************************************
+* Macros
+***************************************/
+#define XXH_STATIC_ASSERT(c) { enum { XXH_sa = 1/(int)(!!(c)) }; } /* use after variable declarations */
+XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }
+
+
+/* *******************************************************************
+* 32-bit hash functions
+*********************************************************************/
+static const U32 PRIME32_1 = 2654435761U; /* 0b10011110001101110111100110110001 */
+static const U32 PRIME32_2 = 2246822519U; /* 0b10000101111010111100101001110111 */
+static const U32 PRIME32_3 = 3266489917U; /* 0b11000010101100101010111000111101 */
+static const U32 PRIME32_4 = 668265263U; /* 0b00100111110101001110101100101111 */
+static const U32 PRIME32_5 = 374761393U; /* 0b00010110010101100110011110110001 */
+
+static U32 XXH32_round(U32 acc, U32 input)
+{
+ acc += input * PRIME32_2;
+ acc = XXH_rotl32(acc, 13);
+ acc *= PRIME32_1;
+#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE)
+ /* UGLY HACK:
+ * This inline assembly hack forces acc into a normal register. This is the
+ * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop
+ * (pragmas and attributes don't work for some resason) without globally
+ * disabling SSE4.1.
+ *
+ * The reason we want to avoid vectorization is because despite working on
+ * 4 integers at a time, there are multiple factors slowing XXH32 down on
+ * SSE4:
+ * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!)
+ * making it slightly slower to multiply four integers at once compared to four
+ * integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is
+ * still not worth it to go into SSE just to multiply unless doing a long operation.
+ *
+ * - Four instructions are required to rotate,
+ * movqda tmp, v // not required with VEX encoding
+ * pslld tmp, 13 // tmp <<= 13
+ * psrld v, 19 // x >>= 19
+ * por v, tmp // x |= tmp
+ * compared to one for scalar:
+ * roll v, 13 // reliably fast across the board
+ * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason
+ *
+ * - Instruction level parallelism is actually more beneficial here because the
+ * SIMD actually serializes this operation: While v1 is rotating, v2 can load data,
+ * while v3 can multiply. SSE forces them to operate together.
+ *
+ * How this hack works:
+ * __asm__("" // Declare an assembly block but don't declare any instructions
+ * : // However, as an Input/Output Operand,
+ * "+r" // constrain a read/write operand (+) as a general purpose register (r).
+ * (acc) // and set acc as the operand
+ * );
+ *
+ * Because of the 'r', the compiler has promised that seed will be in a
+ * general purpose register and the '+' says that it will be 'read/write',
+ * so it has to assume it has changed. It is like volatile without all the
+ * loads and stores.
+ *
+ * Since the argument has to be in a normal register (not an SSE register),
+ * each time XXH32_round is called, it is impossible to vectorize. */
+ __asm__("" : "+r" (acc));
+#endif
+ return acc;
+}
+
+/* mix all bits */
+static U32 XXH32_avalanche(U32 h32)
+{
+ h32 ^= h32 >> 15;
+ h32 *= PRIME32_2;
+ h32 ^= h32 >> 13;
+ h32 *= PRIME32_3;
+ h32 ^= h32 >> 16;
+ return(h32);
+}
+
+#define XXH_get32bits(p) XXH_readLE32_align(p, align)
+
+static U32
+XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align)
+
+{
+ const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1 \
+ h32 += (*p++) * PRIME32_5; \
+ h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
+
+#define PROCESS4 \
+ h32 += XXH_get32bits(p) * PRIME32_3; \
+ p+=4; \
+ h32 = XXH_rotl32(h32, 17) * PRIME32_4 ;
+
+ switch(len&15) /* or switch(bEnd - p) */
+ {
+ case 12: PROCESS4;
+ /* fallthrough */
+ case 8: PROCESS4;
+ /* fallthrough */
+ case 4: PROCESS4;
+ return XXH32_avalanche(h32);
+
+ case 13: PROCESS4;
+ /* fallthrough */
+ case 9: PROCESS4;
+ /* fallthrough */
+ case 5: PROCESS4;
+ PROCESS1;
+ return XXH32_avalanche(h32);
+
+ case 14: PROCESS4;
+ /* fallthrough */
+ case 10: PROCESS4;
+ /* fallthrough */
+ case 6: PROCESS4;
+ PROCESS1;
+ PROCESS1;
+ return XXH32_avalanche(h32);
+
+ case 15: PROCESS4;
+ /* fallthrough */
+ case 11: PROCESS4;
+ /* fallthrough */
+ case 7: PROCESS4;
+ /* fallthrough */
+ case 3: PROCESS1;
+ /* fallthrough */
+ case 2: PROCESS1;
+ /* fallthrough */
+ case 1: PROCESS1;
+ /* fallthrough */
+ case 0: return XXH32_avalanche(h32);
+ }
+ assert(0);
+ return h32; /* reaching this point is deemed impossible */
+}
+
+XXH_FORCE_INLINE U32
+XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)input;
+ const BYTE* bEnd = p + len;
+ U32 h32;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ if (p==NULL) {
+ len=0;
+ bEnd=p=(const BYTE*)(size_t)16;
+ }
+#endif
+
+ if (len>=16) {
+ const BYTE* const limit = bEnd - 15;
+ U32 v1 = seed + PRIME32_1 + PRIME32_2;
+ U32 v2 = seed + PRIME32_2;
+ U32 v3 = seed + 0;
+ U32 v4 = seed - PRIME32_1;
+
+ do {
+ v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
+ v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
+ v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
+ v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
+ } while (p < limit);
+
+ h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7)
+ + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
+ } else {
+ h32 = seed + PRIME32_5;
+ }
+
+ h32 += (U32)len;
+
+ return XXH32_finalize(h32, p, len&15, align);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
+{
+#if 0
+ /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+ XXH32_state_t state;
+ XXH32_reset(&state, seed);
+ XXH32_update(&state, input, len);
+ return XXH32_digest(&state);
+
+#else
+
+ if (XXH_FORCE_ALIGN_CHECK) {
+ if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */
+ return XXH32_endian_align(input, len, seed, XXH_aligned);
+ } }
+
+ return XXH32_endian_align(input, len, seed, XXH_unaligned);
+#endif
+}
+
+
+
+/*====== Hash streaming ======*/
+
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
+{
+ return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
+{
+ XXH_free(statePtr);
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState)
+{
+ memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
+{
+ XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+ memset(&state, 0, sizeof(state));
+ state.v1 = seed + PRIME32_1 + PRIME32_2;
+ state.v2 = seed + PRIME32_2;
+ state.v3 = seed + 0;
+ state.v4 = seed - PRIME32_1;
+ /* do not write into reserved, planned to be removed in a future version */
+ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API XXH_errorcode
+XXH32_update(XXH32_state_t* state, const void* input, size_t len)
+{
+ if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ return XXH_OK;
+#else
+ return XXH_ERROR;
+#endif
+
+ { const BYTE* p = (const BYTE*)input;
+ const BYTE* const bEnd = p + len;
+
+ state->total_len_32 += (XXH32_hash_t)len;
+ state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16));
+
+ if (state->memsize + len < 16) { /* fill in tmp buffer */
+ XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
+ state->memsize += (XXH32_hash_t)len;
+ return XXH_OK;
+ }
+
+ if (state->memsize) { /* some data left from previous update */
+ XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
+ { const U32* p32 = state->mem32;
+ state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++;
+ state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++;
+ state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++;
+ state->v4 = XXH32_round(state->v4, XXH_readLE32(p32));
+ }
+ p += 16-state->memsize;
+ state->memsize = 0;
+ }
+
+ if (p <= bEnd-16) {
+ const BYTE* const limit = bEnd - 16;
+ U32 v1 = state->v1;
+ U32 v2 = state->v2;
+ U32 v3 = state->v3;
+ U32 v4 = state->v4;
+
+ do {
+ v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4;
+ v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4;
+ v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4;
+ v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4;
+ } while (p<=limit);
+
+ state->v1 = v1;
+ state->v2 = v2;
+ state->v3 = v3;
+ state->v4 = v4;
+ }
+
+ if (p < bEnd) {
+ XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
+ state->memsize = (unsigned)(bEnd-p);
+ }
+ }
+
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state)
+{
+ U32 h32;
+
+ if (state->large_len) {
+ h32 = XXH_rotl32(state->v1, 1)
+ + XXH_rotl32(state->v2, 7)
+ + XXH_rotl32(state->v3, 12)
+ + XXH_rotl32(state->v4, 18);
+ } else {
+ h32 = state->v3 /* == seed */ + PRIME32_5;
+ }
+
+ h32 += state->total_len_32;
+
+ return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned);
+}
+
+
+/*====== Canonical representation ======*/
+
+/*! Default XXH result types are basic unsigned 32 and 64 bits.
+* The canonical representation follows human-readable write convention, aka big-endian (large digits first).
+* These functions allow transformation of hash result into and from its canonical format.
+* This way, hash values can be written into a file or buffer, remaining comparable across different systems.
+*/
+
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
+{
+ XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
+ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
+ memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
+{
+ return XXH_readBE32(src);
+}
+
+
+#ifndef XXH_NO_LONG_LONG
+
+/* *******************************************************************
+* 64-bit hash functions
+*********************************************************************/
+
+/*====== Memory access ======*/
+
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+ typedef uint64_t U64;
+# else
+ /* if compiler doesn't support unsigned long long, replace by another 64-bit type */
+ typedef unsigned long long U64;
+# endif
+#endif
+
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64;
+static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+
+static U64 XXH_read64(const void* memPtr)
+{
+ U64 val;
+ memcpy(&val, memPtr, sizeof(val));
+ return val;
+}
+
+#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+#if defined(_MSC_VER) /* Visual Studio */
+# define XXH_swap64 _byteswap_uint64
+#elif XXH_GCC_VERSION >= 403
+# define XXH_swap64 __builtin_bswap64
+#else
+static U64 XXH_swap64 (U64 x)
+{
+ return ((x << 56) & 0xff00000000000000ULL) |
+ ((x << 40) & 0x00ff000000000000ULL) |
+ ((x << 24) & 0x0000ff0000000000ULL) |
+ ((x << 8) & 0x000000ff00000000ULL) |
+ ((x >> 8) & 0x00000000ff000000ULL) |
+ ((x >> 24) & 0x0000000000ff0000ULL) |
+ ((x >> 40) & 0x000000000000ff00ULL) |
+ ((x >> 56) & 0x00000000000000ffULL);
+}
+#endif
+
+XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
+}
+
+static U64 XXH_readBE64(const void* ptr)
+{
+ return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
+}
+
+XXH_FORCE_INLINE U64
+XXH_readLE64_align(const void* ptr, XXH_alignment align)
+{
+ if (align==XXH_unaligned)
+ return XXH_readLE64(ptr);
+ else
+ return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
+}
+
+
+/*====== xxh64 ======*/
+
+static const U64 PRIME64_1 = 11400714785074694791ULL; /* 0b1001111000110111011110011011000110000101111010111100101010000111 */
+static const U64 PRIME64_2 = 14029467366897019727ULL; /* 0b1100001010110010101011100011110100100111110101001110101101001111 */
+static const U64 PRIME64_3 = 1609587929392839161ULL; /* 0b0001011001010110011001111011000110011110001101110111100111111001 */
+static const U64 PRIME64_4 = 9650029242287828579ULL; /* 0b1000010111101011110010100111011111000010101100101010111001100011 */
+static const U64 PRIME64_5 = 2870177450012600261ULL; /* 0b0010011111010100111010110010111100010110010101100110011111000101 */
+
+static U64 XXH64_round(U64 acc, U64 input)
+{
+ acc += input * PRIME64_2;
+ acc = XXH_rotl64(acc, 31);
+ acc *= PRIME64_1;
+ return acc;
+}
+
+static U64 XXH64_mergeRound(U64 acc, U64 val)
+{
+ val = XXH64_round(0, val);
+ acc ^= val;
+ acc = acc * PRIME64_1 + PRIME64_4;
+ return acc;
+}
+
+static U64 XXH64_avalanche(U64 h64)
+{
+ h64 ^= h64 >> 33;
+ h64 *= PRIME64_2;
+ h64 ^= h64 >> 29;
+ h64 *= PRIME64_3;
+ h64 ^= h64 >> 32;
+ return h64;
+}
+
+
+#define XXH_get64bits(p) XXH_readLE64_align(p, align)
+
+static U64
+XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1_64 \
+ h64 ^= (*p++) * PRIME64_5; \
+ h64 = XXH_rotl64(h64, 11) * PRIME64_1;
+
+#define PROCESS4_64 \
+ h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \
+ p+=4; \
+ h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
+
+#define PROCESS8_64 { \
+ U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \
+ p+=8; \
+ h64 ^= k1; \
+ h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \
+}
+
+ switch(len&31) {
+ case 24: PROCESS8_64;
+ /* fallthrough */
+ case 16: PROCESS8_64;
+ /* fallthrough */
+ case 8: PROCESS8_64;
+ return XXH64_avalanche(h64);
+
+ case 28: PROCESS8_64;
+ /* fallthrough */
+ case 20: PROCESS8_64;
+ /* fallthrough */
+ case 12: PROCESS8_64;
+ /* fallthrough */
+ case 4: PROCESS4_64;
+ return XXH64_avalanche(h64);
+
+ case 25: PROCESS8_64;
+ /* fallthrough */
+ case 17: PROCESS8_64;
+ /* fallthrough */
+ case 9: PROCESS8_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 29: PROCESS8_64;
+ /* fallthrough */
+ case 21: PROCESS8_64;
+ /* fallthrough */
+ case 13: PROCESS8_64;
+ /* fallthrough */
+ case 5: PROCESS4_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 26: PROCESS8_64;
+ /* fallthrough */
+ case 18: PROCESS8_64;
+ /* fallthrough */
+ case 10: PROCESS8_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 30: PROCESS8_64;
+ /* fallthrough */
+ case 22: PROCESS8_64;
+ /* fallthrough */
+ case 14: PROCESS8_64;
+ /* fallthrough */
+ case 6: PROCESS4_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 27: PROCESS8_64;
+ /* fallthrough */
+ case 19: PROCESS8_64;
+ /* fallthrough */
+ case 11: PROCESS8_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ PROCESS1_64;
+ return XXH64_avalanche(h64);
+
+ case 31: PROCESS8_64;
+ /* fallthrough */
+ case 23: PROCESS8_64;
+ /* fallthrough */
+ case 15: PROCESS8_64;
+ /* fallthrough */
+ case 7: PROCESS4_64;
+ /* fallthrough */
+ case 3: PROCESS1_64;
+ /* fallthrough */
+ case 2: PROCESS1_64;
+ /* fallthrough */
+ case 1: PROCESS1_64;
+ /* fallthrough */
+ case 0: return XXH64_avalanche(h64);
+ }
+
+ /* impossible to reach */
+ assert(0);
+ return 0; /* unreachable, but some compilers complain without it */
+}
+
+XXH_FORCE_INLINE U64
+XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align)
+{
+ const BYTE* p = (const BYTE*)input;
+ const BYTE* bEnd = p + len;
+ U64 h64;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ if (p==NULL) {
+ len=0;
+ bEnd=p=(const BYTE*)(size_t)32;
+ }
+#endif
+
+ if (len>=32) {
+ const BYTE* const limit = bEnd - 32;
+ U64 v1 = seed + PRIME64_1 + PRIME64_2;
+ U64 v2 = seed + PRIME64_2;
+ U64 v3 = seed + 0;
+ U64 v4 = seed - PRIME64_1;
+
+ do {
+ v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
+ v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
+ v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
+ v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
+ } while (p<=limit);
+
+ h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+ h64 = XXH64_mergeRound(h64, v1);
+ h64 = XXH64_mergeRound(h64, v2);
+ h64 = XXH64_mergeRound(h64, v3);
+ h64 = XXH64_mergeRound(h64, v4);
+
+ } else {
+ h64 = seed + PRIME64_5;
+ }
+
+ h64 += (U64) len;
+
+ return XXH64_finalize(h64, p, len, align);
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
+{
+#if 0
+ /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+ XXH64_state_t state;
+ XXH64_reset(&state, seed);
+ XXH64_update(&state, input, len);
+ return XXH64_digest(&state);
+
+#else
+
+ if (XXH_FORCE_ALIGN_CHECK) {
+ if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */
+ return XXH64_endian_align(input, len, seed, XXH_aligned);
+ } }
+
+ return XXH64_endian_align(input, len, seed, XXH_unaligned);
+
+#endif
+}
+
+/*====== Hash Streaming ======*/
+
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
+{
+ return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
+{
+ XXH_free(statePtr);
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState)
+{
+ memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
+{
+ XXH64_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+ memset(&state, 0, sizeof(state));
+ state.v1 = seed + PRIME64_1 + PRIME64_2;
+ state.v2 = seed + PRIME64_2;
+ state.v3 = seed + 0;
+ state.v4 = seed - PRIME64_1;
+ /* do not write into reserved, planned to be removed in a future version */
+ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+ return XXH_OK;
+}
+
+XXH_PUBLIC_API XXH_errorcode
+XXH64_update (XXH64_state_t* state, const void* input, size_t len)
+{
+ if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+ return XXH_OK;
+#else
+ return XXH_ERROR;
+#endif
+
+ { const BYTE* p = (const BYTE*)input;
+ const BYTE* const bEnd = p + len;
+
+ state->total_len += len;
+
+ if (state->memsize + len < 32) { /* fill in tmp buffer */
+ XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+ state->memsize += (U32)len;
+ return XXH_OK;
+ }
+
+ if (state->memsize) { /* tmp buffer is full */
+ XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
+ state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0));
+ state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1));
+ state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2));
+ state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3));
+ p += 32-state->memsize;
+ state->memsize = 0;
+ }
+
+ if (p+32 <= bEnd) {
+ const BYTE* const limit = bEnd - 32;
+ U64 v1 = state->v1;
+ U64 v2 = state->v2;
+ U64 v3 = state->v3;
+ U64 v4 = state->v4;
+
+ do {
+ v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8;
+ v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8;
+ v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8;
+ v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8;
+ } while (p<=limit);
+
+ state->v1 = v1;
+ state->v2 = v2;
+ state->v3 = v3;
+ state->v4 = v4;
+ }
+
+ if (p < bEnd) {
+ XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
+ state->memsize = (unsigned)(bEnd-p);
+ }
+ }
+
+ return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state)
+{
+ U64 h64;
+
+ if (state->total_len >= 32) {
+ U64 const v1 = state->v1;
+ U64 const v2 = state->v2;
+ U64 const v3 = state->v3;
+ U64 const v4 = state->v4;
+
+ h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+ h64 = XXH64_mergeRound(h64, v1);
+ h64 = XXH64_mergeRound(h64, v2);
+ h64 = XXH64_mergeRound(h64, v3);
+ h64 = XXH64_mergeRound(h64, v4);
+ } else {
+ h64 = state->v3 /*seed*/ + PRIME64_5;
+ }
+
+ h64 += (U64) state->total_len;
+
+ return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned);
+}
+
+
+/*====== Canonical representation ======*/
+
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
+{
+ XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
+ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
+ memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
+{
+ return XXH_readBE64(src);
+}
+
+
+
+/* *********************************************************************
+* XXH3
+* New generation hash designed for speed on small keys and vectorization
+************************************************************************ */
+
+/* #include "xxh3.h" */
+
+
+
+#endif /* XXH_NO_LONG_LONG */
diff --git a/crypto/xxhash.h b/crypto/xxhash.h
new file mode 100644
index 000000000000..9ee05e5dc146
--- /dev/null
+++ b/crypto/xxhash.h
@@ -0,0 +1,445 @@
+/*
+ xxHash - Extremely Fast Hash algorithm
+ Header File
+ Copyright (C) 2012-2016, Yann Collet.
+
+ BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with the
+ distribution.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ You can contact the author at :
+ - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* Notice extracted from xxHash homepage :
+
+xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
+It also successfully passes all tests from the SMHasher suite.
+
+Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
+
+Name Speed Q.Score Author
+xxHash 5.4 GB/s 10
+CrapWow 3.2 GB/s 2 Andrew
+MumurHash 3a 2.7 GB/s 10 Austin Appleby
+SpookyHash 2.0 GB/s 10 Bob Jenkins
+SBox 1.4 GB/s 9 Bret Mulvey
+Lookup3 1.2 GB/s 9 Bob Jenkins
+SuperFastHash 1.2 GB/s 1 Paul Hsieh
+CityHash64 1.05 GB/s 10 Pike & Alakuijala
+FNV 0.55 GB/s 5 Fowler, Noll, Vo
+CRC32 0.43 GB/s 9
+MD5-32 0.33 GB/s 10 Ronald L. Rivest
+SHA1-32 0.28 GB/s 10
+
+Q.Score is a measure of quality of the hash function.
+It depends on successfully passing SMHasher test set.
+10 is a perfect score.
+
+A 64-bit version, named XXH64, is available since r35.
+It offers much better speed, but for 64-bit applications only.
+Name Speed on 64 bits Speed on 32 bits
+XXH64 13.8 GB/s 1.9 GB/s
+XXH32 6.8 GB/s 6.0 GB/s
+*/
+
+#ifndef XXHASH_H_5627135585666179
+#define XXHASH_H_5627135585666179 1
+
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+
+/* ****************************
+* Definitions
+******************************/
+#include <stddef.h> /* size_t */
+typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
+
+
+/* ****************************
+ * API modifier
+ ******************************/
+/** XXH_INLINE_ALL (and XXH_PRIVATE_API)
+ * This is useful to include xxhash functions in `static` mode
+ * in order to inline them, and remove their symbol from the public list.
+ * Inlining can offer dramatic performance improvement on small keys.
+ * Methodology :
+ * #define XXH_INLINE_ALL
+ * #include "xxhash.h"
+ * `xxhash.c` is automatically included.
+ * It's not useful to compile and link it as a separate module.
+ */
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+# ifndef XXH_STATIC_LINKING_ONLY
+# define XXH_STATIC_LINKING_ONLY
+# endif
+# if defined(__GNUC__)
+# define XXH_PUBLIC_API static __inline __attribute__((unused))
+# elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
+# define XXH_PUBLIC_API static inline
+# elif defined(_MSC_VER)
+# define XXH_PUBLIC_API static __inline
+# else
+ /* this version may generate warnings for unused static functions */
+# define XXH_PUBLIC_API static
+# endif
+#else
+# if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT))
+# ifdef XXH_EXPORT
+# define XXH_PUBLIC_API __declspec(dllexport)
+# elif XXH_IMPORT
+# define XXH_PUBLIC_API __declspec(dllimport)
+# endif
+# else
+# define XXH_PUBLIC_API /* do nothing */
+# endif
+#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */
+
+/*! XXH_NAMESPACE, aka Namespace Emulation :
+ *
+ * If you want to include _and expose_ xxHash functions from within your own library,
+ * but also want to avoid symbol collisions with other libraries which may also include xxHash,
+ *
+ * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
+ * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values).
+ *
+ * Note that no change is required within the calling program as long as it includes `xxhash.h` :
+ * regular symbol name will be automatically translated by this header.
+ */
+#ifdef XXH_NAMESPACE
+# define XXH_CAT(A,B) A##B
+# define XXH_NAME2(A,B) XXH_CAT(A,B)
+# define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
+# define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
+# define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
+# define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
+# define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
+# define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
+# define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
+# define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
+# define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
+# define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
+# define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
+# define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
+# define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
+# define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
+# define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
+# define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
+# define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
+# define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
+# define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
+#endif
+
+
+/* *************************************
+* Version
+***************************************/
+#define XXH_VERSION_MAJOR 0
+#define XXH_VERSION_MINOR 7
+#define XXH_VERSION_RELEASE 0
+#define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
+XXH_PUBLIC_API unsigned XXH_versionNumber (void);
+
+
+/*-**********************************************************************
+* 32-bit hash
+************************************************************************/
+typedef unsigned int XXH32_hash_t;
+
+/*! XXH32() :
+ Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input".
+ The memory between input & input+length must be valid (allocated and read-accessible).
+ "seed" can be used to alter the result predictably.
+ Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */
+XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
+
+/*====== Streaming ======*/
+typedef struct XXH32_state_s XXH32_state_t; /* incomplete type */
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr);
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed);
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr);
+
+/*
+ * Streaming functions generate the xxHash of an input provided in multiple segments.
+ * Note that, for small input, they are slower than single-call functions, due to state management.
+ * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized.
+ *
+ * XXH state must first be allocated, using XXH*_createState() .
+ *
+ * Start a new hash by initializing state with a seed, using XXH*_reset().
+ *
+ * Then, feed the hash state by calling XXH*_update() as many times as necessary.
+ * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
+ *
+ * Finally, a hash value can be produced anytime, by using XXH*_digest().
+ * This function returns the nn-bits hash as an int or long long.
+ *
+ * It's still possible to continue inserting input into the hash state after a digest,
+ * and generate some new hashes later on, by calling again XXH*_digest().
+ *
+ * When done, free XXH state space if it was allocated dynamically.
+ */
+
+/*====== Canonical representation ======*/
+
+typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
+
+/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
+ * The canonical representation uses human-readable write convention, aka big-endian (large digits first).
+ * These functions allow transformation of hash result into and from its canonical format.
+ * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
+ */
+
+
+#ifndef XXH_NO_LONG_LONG
+/*-**********************************************************************
+* 64-bit hash
+************************************************************************/
+typedef unsigned long long XXH64_hash_t;
+
+/*! XXH64() :
+ Calculate the 64-bit hash of sequence of length "len" stored at memory address "input".
+ "seed" can be used to alter the result predictably.
+ This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark).
+*/
+XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);
+
+/*====== Streaming ======*/
+typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr);
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed);
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr);
+
+/*====== Canonical representation ======*/
+typedef struct { unsigned char digest[8]; } XXH64_canonical_t;
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);
+
+
+#endif /* XXH_NO_LONG_LONG */
+
+
+
+#ifdef XXH_STATIC_LINKING_ONLY
+
+/* ================================================================================================
+ This section contains declarations which are not guaranteed to remain stable.
+ They may change in future versions, becoming incompatible with a different version of the library.
+ These declarations should only be used with static linking.
+ Never use them in association with dynamic linking !
+=================================================================================================== */
+
+/* These definitions are only present to allow
+ * static allocation of XXH state, on stack or in a struct for example.
+ * Never **ever** use members directly. */
+
+#if !defined (__VMS) \
+ && (defined (__cplusplus) \
+ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+# include <stdint.h>
+
+struct XXH32_state_s {
+ uint32_t total_len_32;
+ uint32_t large_len;
+ uint32_t v1;
+ uint32_t v2;
+ uint32_t v3;
+ uint32_t v4;
+ uint32_t mem32[4];
+ uint32_t memsize;
+ uint32_t reserved; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH32_state_t */
+
+struct XXH64_state_s {
+ uint64_t total_len;
+ uint64_t v1;
+ uint64_t v2;
+ uint64_t v3;
+ uint64_t v4;
+ uint64_t mem64[4];
+ uint32_t memsize;
+ uint32_t reserved[2]; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH64_state_t */
+
+# else
+
+struct XXH32_state_s {
+ XXH32_hash_t total_len_32;
+ XXH32_hash_t large_len;
+ XXH32_hash_t v1;
+ XXH32_hash_t v2;
+ XXH32_hash_t v3;
+ XXH32_hash_t v4;
+ XXH32_hash_t mem32[4];
+ XXH32_hash_t memsize;
+ XXH32_hash_t reserved; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH32_state_t */
+
+# ifndef XXH_NO_LONG_LONG /* remove 64-bit support */
+struct XXH64_state_s {
+ XXH64_hash_t total_len;
+ XXH64_hash_t v1;
+ XXH64_hash_t v2;
+ XXH64_hash_t v3;
+ XXH64_hash_t v4;
+ XXH64_hash_t mem64[4];
+ XXH32_hash_t memsize;
+ XXH32_hash_t reserved[2]; /* never read nor write, might be removed in a future version */
+}; /* typedef'd to XXH64_state_t */
+# endif
+
+# endif
+
+
+/*-**********************************************************************
+* XXH3
+* New experimental hash
+************************************************************************/
+#ifndef XXH_NO_LONG_LONG
+
+
+/* ============================================
+ * XXH3 is a new hash algorithm,
+ * featuring vastly improved speed performance
+ * for both small and large inputs.
+ * A full speed analysis will be published,
+ * it requires a lot more space than this comment can handle.
+ * In general, expect XXH3 to run about ~2x faster on large inputs,
+ * and >3x faster on small ones, though exact difference depend on platform.
+ *
+ * The algorithm is portable, will generate the same hash on all platforms.
+ * It benefits greatly from vectorization units, but does not require it.
+ *
+ * XXH3 offers 2 variants, _64bits and _128bits.
+ * When only 64 bits are needed, prefer calling the _64bits variant :
+ * it reduces the amount of mixing, resulting in faster speed on small inputs.
+ * It's also generally simpler to manipulate a scalar type than a struct.
+ * Note : the low 64-bit field of the _128bits variant is the same as _64bits result.
+ *
+ * The XXH3 algorithm is still considered experimental.
+ * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use.
+ * While labelled experimental, the produced result can still change between versions.
+ *
+ * The API currently supports one-shot hashing only.
+ * The full version will include streaming capability, and canonical representation
+ * Long term optional feature may include custom secret keys, and secret key generation.
+ *
+ * There are still a number of opened questions that community can influence during the experimental period.
+ * I'm trying to list a few of them below, though don't consider this list as complete.
+ *
+ * - 128-bits output type : currently defined as a structure of 2 64-bits fields.
+ * That's because 128-bit values do not exist in C standard.
+ * Note that it means that, at byte level, result is not identical depending on endianess.
+ * However, at field level, they are identical on all platforms.
+ * The canonical representation will solve the issue of identical byte-level representation across platforms,
+ * which is necessary for serialization.
+ * Would there be a better representation for a 128-bit hash result ?
+ * Are the names of the inner 64-bit fields important ? Should they be changed ?
+ *
+ * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian).
+ * What should it be for the 128-bit variant ?
+ * Since it's no longer a scalar value, big-endian representation is no longer an obvious choice.
+ * One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian)
+ * Another one : represent it in the same order as natural order in the struct for little-endian platforms.
+ * Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms.
+ *
+ * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct.
+ * Granted, it's not terribly difficult to create a comparator, but it's still a workload.
+ * Would it be beneficial to declare and define a comparator function for XXH128_hash_t ?
+ * Are there other operations on XXH128_hash_t which would be desirable ?
+ *
+ * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits.
+ * This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases.
+ * But it might influence the design of XXH128_hash_t, in ways which may block other possibilities.
+ * Good idea, bad idea ?
+ *
+ * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant.
+ * It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash.
+ * Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value.
+ * It would either replace current choice, or add a new one.
+ * Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`).
+ * If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ?
+ *
+ * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed.
+ * This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty.
+ * Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64).
+ * But is it still fine to return the seed when the seed is non-zero ?
+ * Are there use case which would depend on this behavior, or would prefer a mixing of the seed ?
+ */
+
+#ifdef XXH_NAMESPACE
+# define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128)
+# define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits)
+# define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed)
+# define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits)
+# define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed)
+#endif
+
+
+typedef struct {
+ XXH64_hash_t low64;
+ XXH64_hash_t high64;
+} XXH128_hash_t;
+
+XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed);
+
+/* note : variants without seed produce same result as variant with seed == 0 */
+XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed); /* == XXH128() */
+
+
+#endif /* XXH_NO_LONG_LONG */
+
+
+/*-**********************************************************************
+* XXH_INLINE_ALL
+************************************************************************/
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+# include "xxhash.c" /* include xxhash function bodies as `static`, for inlining */
+#endif
+
+
+
+#endif /* XXH_STATIC_LINKING_ONLY */
+
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* XXHASH_H_5627135585666179 */
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (8 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 09/12] btrfs-progs: add xxhash sources Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-04 8:31 ` Nikolay Borisov
2019-09-24 14:23 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
` (2 subsequent siblings)
12 siblings, 2 replies; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
From: David Sterba <dsterba@suse.com>
Add xxhash64 as another checksumming algorithm.
Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Changes to v3:
- Fix usage of is_valid_csum_type() (Nikolay)
- Remove unrelated whitespace change (Nikolay)
Changes to v2:
- Integrated comments from Nikolay
---
Makefile | 3 ++-
cmds/inspect-dump-super.c | 24 +++++++++++++++---------
convert/common.c | 2 +-
convert/main.c | 2 +-
crypto/hash.c | 16 ++++++++++++++++
crypto/hash.h | 10 ++++++++++
ctree.h | 14 ++++++++++----
disk-io.c | 7 +++++--
image/main.c | 5 +++--
mkfs/common.c | 14 +++++++-------
mkfs/main.c | 6 +++++-
11 files changed, 75 insertions(+), 28 deletions(-)
create mode 100644 crypto/hash.c
create mode 100644 crypto/hash.h
diff --git a/Makefile b/Makefile
index 370e0c37ff65..45530749e2b9 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
kernel-lib/crc32c.o common/messages.o \
- uuid-tree.o utils-lib.o common/rbtree-utils.o
+ uuid-tree.o utils-lib.o common/rbtree-utils.o \
+ crypto/hash.o crypto/xxhash.o
libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 58bf82b0bbd3..f9f38751f429 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
super_flags_num, BTRFS_SUPER_FLAG_SUPP);
}
+static bool is_valid_csum_type(u16 csum_type)
+{
+ switch (csum_type) {
+ case BTRFS_CSUM_TYPE_CRC32:
+ case BTRFS_CSUM_TYPE_XXHASH:
+ return true;
+ default:
+ return false;
+ }
+}
+
static void dump_superblock(struct btrfs_super_block *sb, int full)
{
int i;
@@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
csum_type = btrfs_super_csum_type(sb);
csum_size = BTRFS_CSUM_SIZE;
printf("csum_type\t\t%hu (", csum_type);
- if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+ if (!is_valid_csum_type(csum_type)) {
printf("INVALID");
} else {
- if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
- printf("crc32c");
- csum_size = btrfs_csum_sizes[csum_type];
- } else {
- printf("unknown");
- }
+ printf("%s", btrfs_csums[csum_type].name);
+ csum_size = btrfs_csums[csum_type].size;
}
printf(")\n");
printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
@@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
printf("csum\t\t\t0x");
for (i = 0, p = sb->csum; i < csum_size; i++)
printf("%02x", p[i]);
- if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
- csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
+ if (!is_valid_csum_type(csum_type))
printf(" [UNKNOWN CSUM TYPE OR SIZE]");
else if (check_csum_sblock(sb, csum_size, csum_type))
printf(" [match]");
diff --git a/convert/common.c b/convert/common.c
index 2e2318a5863e..5dd1a2644bf6 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
{
int ret;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
/* Temporary extent buffer is always mapped 1:1 on disk */
diff --git a/convert/main.c b/convert/main.c
index 5e6b12431f59..5eb2a59fb68a 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
- csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+ csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
btrfs_super_csum_type(super));
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
BTRFS_SUPER_INFO_OFFSET);
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..fda7fc4e9f23
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,16 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+ XXH64_hash_t hash;
+
+ hash = XXH64(buf, length, 0);
+ /* NOTE: we're not taking the canonical form here but the plain hash to
+ * be compatible with the kernel implementation!
+ */
+ memcpy(out, &hash, 8);
+
+ return 0;
+}
+
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX 32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/ctree.h b/ctree.h
index 870d9f4948de..4ded8161d149 100644
--- a/ctree.h
+++ b/ctree.h
@@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
/* csum types */
enum btrfs_csum_type {
BTRFS_CSUM_TYPE_CRC32 = 0,
+ BTRFS_CSUM_TYPE_XXHASH = 1,
};
-/* four bytes for CRC32 */
-static int btrfs_csum_sizes[] = { 4 };
+static struct btrfs_csum {
+ u16 size;
+ const char *name;
+} btrfs_csums[] = {
+ [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+ [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
+};
#define BTRFS_EMPTY_DIR_SIZE 0
@@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
{
int t = btrfs_super_csum_type(s);
- BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
- return btrfs_csum_sizes[t];
+ BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
+ return btrfs_csums[t].size;
}
static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
diff --git a/disk-io.c b/disk-io.c
index 810c2e14294a..ce0b746f4db9 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
#include "print-tree.h"
#include "common/rbtree-utils.h"
#include "common/device-scan.h"
+#include "crypto/hash.h"
/* specified errno for check_tree_block */
#define BTRFS_BAD_BYTENR (-1)
@@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
crc = crc32c(crc, data, len);
put_unaligned_le32(~crc, out);
return 0;
+ case BTRFS_CSUM_TYPE_XXHASH:
+ return hash_xxhash(data, len, out);
default:
fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
ASSERT(0);
@@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
}
csum_type = btrfs_super_csum_type(sb);
- if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+ if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
error("unsupported checksum algorithm %u", csum_type);
return -EIO;
}
- csum_size = btrfs_csum_sizes[csum_type];
+ csum_size = btrfs_csums[csum_type].size;
btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
diff --git a/image/main.c b/image/main.c
index 0c8ffede56f5..1265152cf524 100644
--- a/image/main.c
+++ b/image/main.c
@@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
static void csum_block(u8 *buf, size_t len)
{
- u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
+ u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
+ u8 result[csum_size];
u32 crc = ~(u32)0;
crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
put_unaligned_le32(~crc, result);
- memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
+ memcpy(buf, result, csum_size);
}
static int has_name(struct btrfs_key *key)
diff --git a/mkfs/common.c b/mkfs/common.c
index 4a417bd7a306..939be5eb2dc2 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
}
/* generate checksum */
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
/* write back root tree */
@@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
if (ret != cfg->nodesize) {
@@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
if (ret != cfg->nodesize) {
@@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
btrfs_set_header_nritems(buf, nritems);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
if (ret != cfg->nodesize) {
@@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
if (ret != cfg->nodesize) {
@@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
btrfs_set_header_nritems(buf, 0);
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
if (ret != cfg->nodesize) {
@@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
memcpy(buf->data, &super, sizeof(super));
buf->len = BTRFS_SUPER_INFO_SIZE;
- csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+ csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
cfg->csum_type);
ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
cfg->blocks[MKFS_SUPER_BLOCK]);
diff --git a/mkfs/main.c b/mkfs/main.c
index e96cbc5399a2..64806dac7706 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
{
if (strcasecmp(s, "crc32c") == 0) {
return BTRFS_CSUM_TYPE_CRC32;
+ } else if (strcasecmp(s, "xxhash64") == 0 ||
+ strcasecmp(s, "xxhash") == 0) {
+ return BTRFS_CSUM_TYPE_XXHASH;
} else {
error("unknown csum type %s", s);
exit(1);
@@ -1376,7 +1379,8 @@ raid_groups:
pretty_size(allocation.system));
printf("SSD detected: %s\n", ssd ? "yes" : "no");
btrfs_parse_features_to_string(features_buf, features);
- printf("Incompat features: %s", features_buf);
+ printf("Incompat features: %s\n", features_buf);
+ printf("Checksum: %s", btrfs_csums[csum_type].name);
printf("\n");
list_all_devices(root);
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (9 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-04 8:32 ` Nikolay Borisov
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
2019-09-24 14:34 ` [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums David Sterba
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
With the introduction of xxhash64 to btrfs-progs we created a crypto/
directory for all the hashes used in btrfs (although no
cryptographically secure hash is there yet).
Move the crc32c implementation from kernel-lib/ to crypto/ as well so we
have all hashes consolidated.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
Android.mk | 4 ++--
Makefile | 4 ++--
btrfs-crc.c | 2 +-
btrfs-find-root.c | 2 +-
btrfs-sb-mod.c | 2 +-
btrfs.c | 2 +-
cmds/inspect-dump-super.c | 2 +-
cmds/rescue-chunk-recover.c | 2 +-
cmds/rescue-super-recover.c | 2 +-
common/utils.c | 2 +-
convert/main.c | 2 +-
{kernel-lib => crypto}/crc32c.c | 2 +-
{kernel-lib => crypto}/crc32c.h | 0
disk-io.c | 2 +-
extent-tree.c | 2 +-
file-item.c | 2 +-
free-space-cache.c | 2 +-
hash.h | 2 +-
image/main.c | 2 +-
image/sanitize.c | 2 +-
library-test.c | 2 +-
mkfs/main.c | 2 +-
send-stream.c | 2 +-
23 files changed, 24 insertions(+), 24 deletions(-)
rename {kernel-lib => crypto}/crc32c.c (99%)
rename {kernel-lib => crypto}/crc32c.h (100%)
diff --git a/Android.mk b/Android.mk
index e8de47eb4617..8288ba7356f4 100644
--- a/Android.mk
+++ b/Android.mk
@@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \
cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \
mkfs/common.c
libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \
- kernel-lib/crc32c.c messages.c \
+ crypto/crc32c.c messages.c \
uuid-tree.c utils-lib.c rbtree-utils.c
libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
- kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+ crypto/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
blkid_objects := partition/ superblocks/ topology/
diff --git a/Makefile b/Makefile
index 45530749e2b9..c241c22d1018 100644
--- a/Makefile
+++ b/Makefile
@@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
- kernel-lib/crc32c.o common/messages.o \
+ crypto/crc32c.o common/messages.o \
uuid-tree.o utils-lib.o common/rbtree-utils.o \
crypto/hash.o crypto/xxhash.o
libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
- kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
+ crypto/crc32c.h kernel-lib/list.h kerncompat.h \
kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h)
diff --git a/btrfs-crc.c b/btrfs-crc.c
index bcf25df8b46a..c4f81fc65f67 100644
--- a/btrfs-crc.c
+++ b/btrfs-crc.c
@@ -19,7 +19,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
void print_usage(int status)
diff --git a/btrfs-find-root.c b/btrfs-find-root.c
index e46fa8723b33..741eb9a95ac5 100644
--- a/btrfs-find-root.c
+++ b/btrfs-find-root.c
@@ -32,7 +32,7 @@
#include "kernel-lib/list.h"
#include "volumes.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "extent-cache.h"
#include "find-root.h"
#include "common/help.h"
diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 105b556b0cf1..348991b39451 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -24,7 +24,7 @@
#include <string.h>
#include <limits.h>
#include <byteswap.h>
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "disk-io.h"
#define BLOCKSIZE (4096)
diff --git a/btrfs.c b/btrfs.c
index 6c8aabe24dc8..72dad6fb3983 100644
--- a/btrfs.c
+++ b/btrfs.c
@@ -20,7 +20,7 @@
#include <getopt.h>
#include "volumes.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "cmds/commands.h"
#include "common/utils.h"
#include "common/help.h"
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index f9f38751f429..ef3d9094e661 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -32,7 +32,7 @@
#include "kernel-lib/list.h"
#include "common/utils.h"
#include "cmds/commands.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/help.h"
static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 329a608dfc6b..bf35693ddbfa 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -36,7 +36,7 @@
#include "disk-io.h"
#include "volumes.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "check/common.h"
#include "cmds/commands.h"
diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c
index ea3a00caf56c..5d6bea836c8b 100644
--- a/cmds/rescue-super-recover.c
+++ b/cmds/rescue-super-recover.c
@@ -31,7 +31,7 @@
#include "disk-io.h"
#include "kernel-lib/list.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "volumes.h"
#include "cmds/commands.h"
#include "cmds/rescue.h"
diff --git a/common/utils.c b/common/utils.c
index f2a10cccca86..fa49c01ad102 100644
--- a/common/utils.c
+++ b/common/utils.c
@@ -47,7 +47,7 @@
#include "ctree.h"
#include "disk-io.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "common/path-utils.h"
#include "common/device-scan.h"
diff --git a/convert/main.c b/convert/main.c
index 5eb2a59fb68a..fdbddc846e39 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -102,7 +102,7 @@
#include "mkfs/common.h"
#include "convert/common.h"
#include "convert/source-fs.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/fsfeatures.h"
#include "common/box.h"
diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c
similarity index 99%
rename from kernel-lib/crc32c.c
rename to crypto/crc32c.c
index 36bb6f189971..bd6283d5baeb 100644
--- a/kernel-lib/crc32c.c
+++ b/crypto/crc32c.c
@@ -8,7 +8,7 @@
*
*/
#include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include <inttypes.h>
#include <string.h>
#include <unistd.h>
diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h
similarity index 100%
rename from kernel-lib/crc32c.h
rename to crypto/crc32c.h
diff --git a/disk-io.c b/disk-io.c
index ce0b746f4db9..4093982cf3dc 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -29,7 +29,7 @@
#include "disk-io.h"
#include "volumes.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
#include "print-tree.h"
#include "common/rbtree-utils.h"
diff --git a/extent-tree.c b/extent-tree.c
index 932af2c644bd..a8f57776bd73 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -26,7 +26,7 @@
#include "disk-io.h"
#include "print-tree.h"
#include "transaction.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "volumes.h"
#include "free-space-cache.h"
#include "free-space-tree.h"
diff --git a/file-item.c b/file-item.c
index c6e9d212bcab..64af57693baf 100644
--- a/file-item.c
+++ b/file-item.c
@@ -24,7 +24,7 @@
#include "disk-io.h"
#include "transaction.h"
#include "print-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/internal.h"
#define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
diff --git a/free-space-cache.c b/free-space-cache.c
index 8a57f86dc650..6e7d7e1ef561 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -22,7 +22,7 @@
#include "transaction.h"
#include "disk-io.h"
#include "extent_io.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "kernel-lib/bitops.h"
#include "common/internal.h"
#include "common/utils.h"
diff --git a/hash.h b/hash.h
index 5398e8869316..51e842047093 100644
--- a/hash.h
+++ b/hash.h
@@ -19,7 +19,7 @@
#ifndef __BTRFS_HASH_H__
#define __BTRFS_HASH_H__
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
static inline u64 btrfs_name_hash(const char *name, int len)
{
diff --git a/image/main.c b/image/main.c
index 1265152cf524..92ee28d7fe1a 100644
--- a/image/main.c
+++ b/image/main.c
@@ -28,7 +28,7 @@
#include <getopt.h>
#include "kerncompat.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "ctree.h"
#include "disk-io.h"
#include "transaction.h"
diff --git a/image/sanitize.c b/image/sanitize.c
index 9caa5deaf2dd..cd2bd6fe2379 100644
--- a/image/sanitize.c
+++ b/image/sanitize.c
@@ -18,7 +18,7 @@
#include "common/internal.h"
#include "common/messages.h"
#include "common/utils.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "image/sanitize.h"
#include "extent_io.h"
diff --git a/library-test.c b/library-test.c
index c44bad228e50..e47917c25830 100644
--- a/library-test.c
+++ b/library-test.c
@@ -21,7 +21,7 @@
#include "version.h"
#include "kernel-lib/rbtree.h"
#include "kernel-lib/radix-tree.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "kernel-lib/list.h"
#include "kernel-lib/sizes.h"
#include "ctree.h"
diff --git a/mkfs/main.c b/mkfs/main.c
index 64806dac7706..a46205e7237f 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -45,7 +45,7 @@
#include "common/rbtree-utils.h"
#include "mkfs/common.h"
#include "mkfs/rootdir.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/fsfeatures.h"
#include "common/box.h"
diff --git a/send-stream.c b/send-stream.c
index 08687e5e6904..3b8ada2110aa 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -21,7 +21,7 @@
#include "send.h"
#include "send-stream.h"
-#include "kernel-lib/crc32c.h"
+#include "crypto/crc32c.h"
#include "common/utils.h"
struct btrfs_send_stream {
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (10 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
@ 2019-09-03 15:00 ` Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-24 14:34 ` [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums David Sterba
12 siblings, 1 reply; 23+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 15:00 UTC (permalink / raw)
To: David Sterba; +Cc: Linux BTRFS Mailinglist, Johannes Thumshirn
Add test-cases for creating a file-system xxhash64 as checksumming
algorithm.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 6e295274119d..3fa3c8ad42d1 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -46,6 +46,8 @@ test_mkfs_single -d single -m dup
test_mkfs_single -d dup -m single
test_mkfs_single -d dup -m dup
test_mkfs_single -d dup -m dup --mixed
+test_mkfs_single -C xxhash64
+test_mkfs_single -C xxhash
test_mkfs_multi
test_mkfs_multi -d single -m single
--
2.16.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
@ 2019-09-04 8:31 ` Nikolay Borisov
2019-09-24 14:23 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: Nikolay Borisov @ 2019-09-04 8:31 UTC (permalink / raw)
To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist
On 3.09.19 г. 18:00 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
>
> Add xxhash64 as another checksumming algorithm.
>
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
>
> ---
> Changes to v3:
> - Fix usage of is_valid_csum_type() (Nikolay)
> - Remove unrelated whitespace change (Nikolay)
>
> Changes to v2:
> - Integrated comments from Nikolay
> ---
> Makefile | 3 ++-
> cmds/inspect-dump-super.c | 24 +++++++++++++++---------
> convert/common.c | 2 +-
> convert/main.c | 2 +-
> crypto/hash.c | 16 ++++++++++++++++
> crypto/hash.h | 10 ++++++++++
> ctree.h | 14 ++++++++++----
> disk-io.c | 7 +++++--
> image/main.c | 5 +++--
> mkfs/common.c | 14 +++++++-------
> mkfs/main.c | 6 +++++-
> 11 files changed, 75 insertions(+), 28 deletions(-)
> create mode 100644 crypto/hash.c
> create mode 100644 crypto/hash.h
>
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> kernel-lib/crc32c.o common/messages.o \
> - uuid-tree.o utils-lib.o common/rbtree-utils.o
> + uuid-tree.o utils-lib.o common/rbtree-utils.o \
> + crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 58bf82b0bbd3..f9f38751f429 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
> super_flags_num, BTRFS_SUPER_FLAG_SUPP);
> }
>
> +static bool is_valid_csum_type(u16 csum_type)
> +{
> + switch (csum_type) {
> + case BTRFS_CSUM_TYPE_CRC32:
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> static void dump_superblock(struct btrfs_super_block *sb, int full)
> {
> int i;
> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> csum_type = btrfs_super_csum_type(sb);
> csum_size = BTRFS_CSUM_SIZE;
> printf("csum_type\t\t%hu (", csum_type);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (!is_valid_csum_type(csum_type)) {
> printf("INVALID");
> } else {
> - if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
> - printf("crc32c");
> - csum_size = btrfs_csum_sizes[csum_type];
> - } else {
> - printf("unknown");
> - }
> + printf("%s", btrfs_csums[csum_type].name);
> + csum_size = btrfs_csums[csum_type].size;
> }
> printf(")\n");
> printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
> @@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> printf("csum\t\t\t0x");
> for (i = 0, p = sb->csum; i < csum_size; i++)
> printf("%02x", p[i]);
> - if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> + if (!is_valid_csum_type(csum_type))
> printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> else if (check_csum_sblock(sb, csum_size, csum_type))
> printf(" [match]");
> diff --git a/convert/common.c b/convert/common.c
> index 2e2318a5863e..5dd1a2644bf6 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
> {
> int ret;
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* Temporary extent buffer is always mapped 1:1 on disk */
> diff --git a/convert/main.c b/convert/main.c
> index 5e6b12431f59..5eb2a59fb68a 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
> BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
> btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
> btrfs_super_csum_type(super));
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> BTRFS_SUPER_INFO_OFFSET);
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..fda7fc4e9f23
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> + XXH64_hash_t hash;
> +
> + hash = XXH64(buf, length, 0);
> + /* NOTE: we're not taking the canonical form here but the plain hash to
> + * be compatible with the kernel implementation!
> + */
> + memcpy(out, &hash, 8);
> +
> + return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX 32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.h b/ctree.h
> index 870d9f4948de..4ded8161d149 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
> /* csum types */
> enum btrfs_csum_type {
> BTRFS_CSUM_TYPE_CRC32 = 0,
> + BTRFS_CSUM_TYPE_XXHASH = 1,
> };
>
> -/* four bytes for CRC32 */
> -static int btrfs_csum_sizes[] = { 4 };
> +static struct btrfs_csum {
> + u16 size;
> + const char *name;
> +} btrfs_csums[] = {
> + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
> +};
>
> #define BTRFS_EMPTY_DIR_SIZE 0
>
> @@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
> static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
> {
> int t = btrfs_super_csum_type(s);
> - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
> - return btrfs_csum_sizes[t];
> + BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
> + return btrfs_csums[t].size;
> }
>
> static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
> diff --git a/disk-io.c b/disk-io.c
> index 810c2e14294a..ce0b746f4db9 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> #include "common/device-scan.h"
> +#include "crypto/hash.h"
>
> /* specified errno for check_tree_block */
> #define BTRFS_BAD_BYTENR (-1)
> @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
> crc = crc32c(crc, data, len);
> put_unaligned_le32(~crc, out);
> return 0;
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return hash_xxhash(data, len, out);
> default:
> fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
> ASSERT(0);
> @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
> }
>
> csum_type = btrfs_super_csum_type(sb);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
> error("unsupported checksum algorithm %u", csum_type);
> return -EIO;
> }
> - csum_size = btrfs_csum_sizes[csum_type];
> + csum_size = btrfs_csums[csum_type].size;
>
> btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
> result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> diff --git a/image/main.c b/image/main.c
> index 0c8ffede56f5..1265152cf524 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
>
> static void csum_block(u8 *buf, size_t len)
> {
> - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
> + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
> + u8 result[csum_size];
> u32 crc = ~(u32)0;
> crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> put_unaligned_le32(~crc, result);
> - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
> + memcpy(buf, result, csum_size);
> }
>
> static int has_name(struct btrfs_key *key)
> diff --git a/mkfs/common.c b/mkfs/common.c
> index 4a417bd7a306..939be5eb2dc2 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* write back root tree */
> @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
> btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
> if (ret != cfg->nodesize) {
> @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
> if (ret != cfg->nodesize) {
> @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
> btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
> if (ret != cfg->nodesize) {
> @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
> btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
> if (ret != cfg->nodesize) {
> @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
> if (ret != cfg->nodesize) {
> @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
> memcpy(buf->data, &super, sizeof(super));
> buf->len = BTRFS_SUPER_INFO_SIZE;
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> cfg->blocks[MKFS_SUPER_BLOCK]);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index e96cbc5399a2..64806dac7706 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> {
> if (strcasecmp(s, "crc32c") == 0) {
> return BTRFS_CSUM_TYPE_CRC32;
> + } else if (strcasecmp(s, "xxhash64") == 0 ||
> + strcasecmp(s, "xxhash") == 0) {
> + return BTRFS_CSUM_TYPE_XXHASH;
> } else {
> error("unknown csum type %s", s);
> exit(1);
> @@ -1376,7 +1379,8 @@ raid_groups:
> pretty_size(allocation.system));
> printf("SSD detected: %s\n", ssd ? "yes" : "no");
> btrfs_parse_features_to_string(features_buf, features);
> - printf("Incompat features: %s", features_buf);
> + printf("Incompat features: %s\n", features_buf);
> + printf("Checksum: %s", btrfs_csums[csum_type].name);
> printf("\n");
>
> list_all_devices(root);
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
@ 2019-09-04 8:32 ` Nikolay Borisov
0 siblings, 0 replies; 23+ messages in thread
From: Nikolay Borisov @ 2019-09-04 8:32 UTC (permalink / raw)
To: Johannes Thumshirn, David Sterba; +Cc: Linux BTRFS Mailinglist
On 3.09.19 г. 18:00 ч., Johannes Thumshirn wrote:
> With the introduction of xxhash64 to btrfs-progs we created a crypto/
> directory for all the hashes used in btrfs (although no
> cryptographically secure hash is there yet).
>
> Move the crc32c implementation from kernel-lib/ to crypto/ as well so we
> have all hashes consolidated.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> Android.mk | 4 ++--
> Makefile | 4 ++--
> btrfs-crc.c | 2 +-
> btrfs-find-root.c | 2 +-
> btrfs-sb-mod.c | 2 +-
> btrfs.c | 2 +-
> cmds/inspect-dump-super.c | 2 +-
> cmds/rescue-chunk-recover.c | 2 +-
> cmds/rescue-super-recover.c | 2 +-
> common/utils.c | 2 +-
> convert/main.c | 2 +-
> {kernel-lib => crypto}/crc32c.c | 2 +-
> {kernel-lib => crypto}/crc32c.h | 0
> disk-io.c | 2 +-
> extent-tree.c | 2 +-
> file-item.c | 2 +-
> free-space-cache.c | 2 +-
> hash.h | 2 +-
> image/main.c | 2 +-
> image/sanitize.c | 2 +-
> library-test.c | 2 +-
> mkfs/main.c | 2 +-
> send-stream.c | 2 +-
> 23 files changed, 24 insertions(+), 24 deletions(-)
> rename {kernel-lib => crypto}/crc32c.c (99%)
> rename {kernel-lib => crypto}/crc32c.h (100%)
>
> diff --git a/Android.mk b/Android.mk
> index e8de47eb4617..8288ba7356f4 100644
> --- a/Android.mk
> +++ b/Android.mk
> @@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \
> cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \
> mkfs/common.c
> libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \
> - kernel-lib/crc32c.c messages.c \
> + crypto/crc32c.c messages.c \
> uuid-tree.c utils-lib.c rbtree-utils.c
> libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> + crypto/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
> blkid_objects := partition/ superblocks/ topology/
> diff --git a/Makefile b/Makefile
> index 45530749e2b9..c241c22d1018 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> - kernel-lib/crc32c.o common/messages.o \
> + crypto/crc32c.o common/messages.o \
> uuid-tree.o utils-lib.o common/rbtree-utils.o \
> crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> + crypto/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
> libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h)
> diff --git a/btrfs-crc.c b/btrfs-crc.c
> index bcf25df8b46a..c4f81fc65f67 100644
> --- a/btrfs-crc.c
> +++ b/btrfs-crc.c
> @@ -19,7 +19,7 @@
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
>
> void print_usage(int status)
> diff --git a/btrfs-find-root.c b/btrfs-find-root.c
> index e46fa8723b33..741eb9a95ac5 100644
> --- a/btrfs-find-root.c
> +++ b/btrfs-find-root.c
> @@ -32,7 +32,7 @@
> #include "kernel-lib/list.h"
> #include "volumes.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "extent-cache.h"
> #include "find-root.h"
> #include "common/help.h"
> diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
> index 105b556b0cf1..348991b39451 100644
> --- a/btrfs-sb-mod.c
> +++ b/btrfs-sb-mod.c
> @@ -24,7 +24,7 @@
> #include <string.h>
> #include <limits.h>
> #include <byteswap.h>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "disk-io.h"
>
> #define BLOCKSIZE (4096)
> diff --git a/btrfs.c b/btrfs.c
> index 6c8aabe24dc8..72dad6fb3983 100644
> --- a/btrfs.c
> +++ b/btrfs.c
> @@ -20,7 +20,7 @@
> #include <getopt.h>
>
> #include "volumes.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "cmds/commands.h"
> #include "common/utils.h"
> #include "common/help.h"
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index f9f38751f429..ef3d9094e661 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -32,7 +32,7 @@
> #include "kernel-lib/list.h"
> #include "common/utils.h"
> #include "cmds/commands.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/help.h"
>
> static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 329a608dfc6b..bf35693ddbfa 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -36,7 +36,7 @@
> #include "disk-io.h"
> #include "volumes.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "check/common.h"
> #include "cmds/commands.h"
> diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c
> index ea3a00caf56c..5d6bea836c8b 100644
> --- a/cmds/rescue-super-recover.c
> +++ b/cmds/rescue-super-recover.c
> @@ -31,7 +31,7 @@
> #include "disk-io.h"
> #include "kernel-lib/list.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "volumes.h"
> #include "cmds/commands.h"
> #include "cmds/rescue.h"
> diff --git a/common/utils.c b/common/utils.c
> index f2a10cccca86..fa49c01ad102 100644
> --- a/common/utils.c
> +++ b/common/utils.c
> @@ -47,7 +47,7 @@
> #include "ctree.h"
> #include "disk-io.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "common/path-utils.h"
> #include "common/device-scan.h"
> diff --git a/convert/main.c b/convert/main.c
> index 5eb2a59fb68a..fdbddc846e39 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -102,7 +102,7 @@
> #include "mkfs/common.h"
> #include "convert/common.h"
> #include "convert/source-fs.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/fsfeatures.h"
> #include "common/box.h"
>
> diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c
> similarity index 99%
> rename from kernel-lib/crc32c.c
> rename to crypto/crc32c.c
> index 36bb6f189971..bd6283d5baeb 100644
> --- a/kernel-lib/crc32c.c
> +++ b/crypto/crc32c.c
> @@ -8,7 +8,7 @@
> *
> */
> #include "kerncompat.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include <inttypes.h>
> #include <string.h>
> #include <unistd.h>
> diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h
> similarity index 100%
> rename from kernel-lib/crc32c.h
> rename to crypto/crc32c.h
> diff --git a/disk-io.c b/disk-io.c
> index ce0b746f4db9..4093982cf3dc 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -29,7 +29,7 @@
> #include "disk-io.h"
> #include "volumes.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> diff --git a/extent-tree.c b/extent-tree.c
> index 932af2c644bd..a8f57776bd73 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -26,7 +26,7 @@
> #include "disk-io.h"
> #include "print-tree.h"
> #include "transaction.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "volumes.h"
> #include "free-space-cache.h"
> #include "free-space-tree.h"
> diff --git a/file-item.c b/file-item.c
> index c6e9d212bcab..64af57693baf 100644
> --- a/file-item.c
> +++ b/file-item.c
> @@ -24,7 +24,7 @@
> #include "disk-io.h"
> #include "transaction.h"
> #include "print-tree.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/internal.h"
>
> #define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
> diff --git a/free-space-cache.c b/free-space-cache.c
> index 8a57f86dc650..6e7d7e1ef561 100644
> --- a/free-space-cache.c
> +++ b/free-space-cache.c
> @@ -22,7 +22,7 @@
> #include "transaction.h"
> #include "disk-io.h"
> #include "extent_io.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "kernel-lib/bitops.h"
> #include "common/internal.h"
> #include "common/utils.h"
> diff --git a/hash.h b/hash.h
> index 5398e8869316..51e842047093 100644
> --- a/hash.h
> +++ b/hash.h
> @@ -19,7 +19,7 @@
> #ifndef __BTRFS_HASH_H__
> #define __BTRFS_HASH_H__
>
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
>
> static inline u64 btrfs_name_hash(const char *name, int len)
> {
> diff --git a/image/main.c b/image/main.c
> index 1265152cf524..92ee28d7fe1a 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -28,7 +28,7 @@
> #include <getopt.h>
>
> #include "kerncompat.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "ctree.h"
> #include "disk-io.h"
> #include "transaction.h"
> diff --git a/image/sanitize.c b/image/sanitize.c
> index 9caa5deaf2dd..cd2bd6fe2379 100644
> --- a/image/sanitize.c
> +++ b/image/sanitize.c
> @@ -18,7 +18,7 @@
> #include "common/internal.h"
> #include "common/messages.h"
> #include "common/utils.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "image/sanitize.h"
> #include "extent_io.h"
>
> diff --git a/library-test.c b/library-test.c
> index c44bad228e50..e47917c25830 100644
> --- a/library-test.c
> +++ b/library-test.c
> @@ -21,7 +21,7 @@
> #include "version.h"
> #include "kernel-lib/rbtree.h"
> #include "kernel-lib/radix-tree.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "kernel-lib/list.h"
> #include "kernel-lib/sizes.h"
> #include "ctree.h"
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 64806dac7706..a46205e7237f 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -45,7 +45,7 @@
> #include "common/rbtree-utils.h"
> #include "mkfs/common.h"
> #include "mkfs/rootdir.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/fsfeatures.h"
> #include "common/box.h"
>
> diff --git a/send-stream.c b/send-stream.c
> index 08687e5e6904..3b8ada2110aa 100644
> --- a/send-stream.c
> +++ b/send-stream.c
> @@ -21,7 +21,7 @@
>
> #include "send.h"
> #include "send-stream.h"
> -#include "kernel-lib/crc32c.h"
> +#include "crypto/crc32c.h"
> #include "common/utils.h"
>
> struct btrfs_send_stream {
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
@ 2019-09-24 12:56 ` David Sterba
2019-09-24 13:06 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 12:56 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> index 28912906d0a9..1ca71a4fcce5 100644
> --- a/mkfs/common.h
> +++ b/mkfs/common.h
> @@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
> u64 features;
> /* Size of the filesystem in bytes */
> u64 num_bytes;
> + /* checksum algorithm to use */
> + enum btrfs_csum_type csum_type;
This is defined in the following patch so the compilation breaks here,
I'll see if reordering 1 and 2 fixes that.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
2019-09-24 12:56 ` David Sterba
@ 2019-09-24 13:06 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 13:06 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> mkfs/common.c | 14 +++++++-------
> mkfs/common.h | 2 ++
> 2 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/mkfs/common.c b/mkfs/common.c
> index caca5e707233..b6e549b19272 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
I don't see where cfg->csum_type is initialized. The tests pass so
there's probably some implicit initialization to 0 that makes it work.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
2019-09-04 8:31 ` Nikolay Borisov
@ 2019-09-24 14:23 ` David Sterba
1 sibling, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:23 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:44PM +0200, Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
>
> Add xxhash64 as another checksumming algorithm.
>
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>
> ---
> Changes to v3:
> - Fix usage of is_valid_csum_type() (Nikolay)
> - Remove unrelated whitespace change (Nikolay)
>
> Changes to v2:
> - Integrated comments from Nikolay
> ---
> Makefile | 3 ++-
> cmds/inspect-dump-super.c | 24 +++++++++++++++---------
> convert/common.c | 2 +-
> convert/main.c | 2 +-
> crypto/hash.c | 16 ++++++++++++++++
> crypto/hash.h | 10 ++++++++++
> ctree.h | 14 ++++++++++----
> disk-io.c | 7 +++++--
> image/main.c | 5 +++--
> mkfs/common.c | 14 +++++++-------
> mkfs/main.c | 6 +++++-
> 11 files changed, 75 insertions(+), 28 deletions(-)
> create mode 100644 crypto/hash.c
> create mode 100644 crypto/hash.h
>
> diff --git a/Makefile b/Makefile
> index 370e0c37ff65..45530749e2b9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
> mkfs/common.o check/mode-common.o check/mode-lowmem.o
> libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
> kernel-lib/crc32c.o common/messages.o \
> - uuid-tree.o utils-lib.o common/rbtree-utils.o
> + uuid-tree.o utils-lib.o common/rbtree-utils.o \
> + crypto/hash.o crypto/xxhash.o
> libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
> kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
> kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 58bf82b0bbd3..f9f38751f429 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
> super_flags_num, BTRFS_SUPER_FLAG_SUPP);
> }
>
> +static bool is_valid_csum_type(u16 csum_type)
Please split generic infrastructure updates to a separate patch, and add
the next hash on top of that.
> +{
> + switch (csum_type) {
> + case BTRFS_CSUM_TYPE_CRC32:
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> static void dump_superblock(struct btrfs_super_block *sb, int full)
> {
> int i;
> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> csum_type = btrfs_super_csum_type(sb);
> csum_size = BTRFS_CSUM_SIZE;
> printf("csum_type\t\t%hu (", csum_type);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (!is_valid_csum_type(csum_type)) {
> printf("INVALID");
> } else {
> - if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
> - printf("crc32c");
> - csum_size = btrfs_csum_sizes[csum_type];
> - } else {
> - printf("unknown");
> - }
> + printf("%s", btrfs_csums[csum_type].name);
> + csum_size = btrfs_csums[csum_type].size;
> }
> printf(")\n");
> printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
> @@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
> printf("csum\t\t\t0x");
> for (i = 0, p = sb->csum; i < csum_size; i++)
> printf("%02x", p[i]);
> - if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> + if (!is_valid_csum_type(csum_type))
> printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> else if (check_csum_sblock(sb, csum_size, csum_type))
> printf(" [match]");
> diff --git a/convert/common.c b/convert/common.c
> index 2e2318a5863e..5dd1a2644bf6 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
> {
> int ret;
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* Temporary extent buffer is always mapped 1:1 on disk */
> diff --git a/convert/main.c b/convert/main.c
> index 5e6b12431f59..5eb2a59fb68a 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
> BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
> btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>
> - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
> btrfs_super_csum_type(super));
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> BTRFS_SUPER_INFO_OFFSET);
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..fda7fc4e9f23
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> + XXH64_hash_t hash;
> +
> + hash = XXH64(buf, length, 0);
> + /* NOTE: we're not taking the canonical form here but the plain hash to
> + * be compatible with the kernel implementation!
> + */
/*
* Comment
*/
> + memcpy(out, &hash, 8);
> +
> + return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX 32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/ctree.h b/ctree.h
> index 870d9f4948de..4ded8161d149 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -167,10 +167,16 @@ struct btrfs_free_space_ctl;
> /* csum types */
> enum btrfs_csum_type {
> BTRFS_CSUM_TYPE_CRC32 = 0,
> + BTRFS_CSUM_TYPE_XXHASH = 1,
> };
>
> -/* four bytes for CRC32 */
> -static int btrfs_csum_sizes[] = { 4 };
> +static struct btrfs_csum {
> + u16 size;
> + const char *name;
> +} btrfs_csums[] = {
> + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
> + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
> +};
This should probably go to .c, same as in kernel so we don't have the
defintion copied in all object files that inclde ctree.h.
> #define BTRFS_EMPTY_DIR_SIZE 0
>
> @@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
> static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
> {
> int t = btrfs_super_csum_type(s);
> - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
> - return btrfs_csum_sizes[t];
> + BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
> + return btrfs_csums[t].size;
> }
>
> static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
> diff --git a/disk-io.c b/disk-io.c
> index 810c2e14294a..ce0b746f4db9 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -34,6 +34,7 @@
> #include "print-tree.h"
> #include "common/rbtree-utils.h"
> #include "common/device-scan.h"
> +#include "crypto/hash.h"
>
> /* specified errno for check_tree_block */
> #define BTRFS_BAD_BYTENR (-1)
> @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
> crc = crc32c(crc, data, len);
> put_unaligned_le32(~crc, out);
> return 0;
> + case BTRFS_CSUM_TYPE_XXHASH:
> + return hash_xxhash(data, len, out);
> default:
> fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
> ASSERT(0);
> @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
> }
>
> csum_type = btrfs_super_csum_type(sb);
> - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> + if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
> error("unsupported checksum algorithm %u", csum_type);
> return -EIO;
> }
> - csum_size = btrfs_csum_sizes[csum_type];
> + csum_size = btrfs_csums[csum_type].size;
>
> btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
> result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> diff --git a/image/main.c b/image/main.c
> index 0c8ffede56f5..1265152cf524 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
>
> static void csum_block(u8 *buf, size_t len)
> {
> - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
> + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
> + u8 result[csum_size];
> u32 crc = ~(u32)0;
> crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> put_unaligned_le32(~crc, result);
> - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
> + memcpy(buf, result, csum_size);
> }
>
> static int has_name(struct btrfs_key *key)
> diff --git a/mkfs/common.c b/mkfs/common.c
> index 4a417bd7a306..939be5eb2dc2 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> }
>
> /* generate checksum */
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
>
> /* write back root tree */
> @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
> btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
> if (ret != cfg->nodesize) {
> @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
> if (ret != cfg->nodesize) {
> @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
> btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, nritems);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
> if (ret != cfg->nodesize) {
> @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
> btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
> if (ret != cfg->nodesize) {
> @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
> btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
> btrfs_set_header_nritems(buf, 0);
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
> if (ret != cfg->nodesize) {
> @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
> memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
> memcpy(buf->data, &super, sizeof(super));
> buf->len = BTRFS_SUPER_INFO_SIZE;
> - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
> cfg->csum_type);
> ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
> cfg->blocks[MKFS_SUPER_BLOCK]);
> diff --git a/mkfs/main.c b/mkfs/main.c
> index e96cbc5399a2..64806dac7706 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> {
> if (strcasecmp(s, "crc32c") == 0) {
> return BTRFS_CSUM_TYPE_CRC32;
> + } else if (strcasecmp(s, "xxhash64") == 0 ||
> + strcasecmp(s, "xxhash") == 0) {
> + return BTRFS_CSUM_TYPE_XXHASH;
> } else {
> error("unknown csum type %s", s);
> exit(1);
> @@ -1376,7 +1379,8 @@ raid_groups:
> pretty_size(allocation.system));
> printf("SSD detected: %s\n", ssd ? "yes" : "no");
> btrfs_parse_features_to_string(features_buf, features);
> - printf("Incompat features: %s", features_buf);
> + printf("Incompat features: %s\n", features_buf);
> + printf("Checksum: %s", btrfs_csums[csum_type].name);
> printf("\n");
>
> list_all_devices(root);
> --
> 2.16.4
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
@ 2019-09-24 14:26 ` David Sterba
0 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:26 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:46PM +0200, Johannes Thumshirn wrote:
> Add test-cases for creating a file-system xxhash64 as checksumming
> algorithm.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
> index 6e295274119d..3fa3c8ad42d1 100755
> --- a/tests/mkfs-tests/001-basic-profiles/test.sh
> +++ b/tests/mkfs-tests/001-basic-profiles/test.sh
> @@ -46,6 +46,8 @@ test_mkfs_single -d single -m dup
> test_mkfs_single -d dup -m single
> test_mkfs_single -d dup -m dup
> test_mkfs_single -d dup -m dup --mixed
> +test_mkfs_single -C xxhash64
> +test_mkfs_single -C xxhash
We'll want to do full test coverage with any of the checksum selected
externally, ie. ina similar way how the 'btrfs check --mode=lowmem' is
done
$ make TEST_ENABLE_OVERRIDE=true TEST_ARGS_CHECK=--mode=lowmem test-check
The support code in the testing sicripts is not there but can be copied
and adapted for mkfs.
A separate test that quickly enumerates over the supported mkfs hashes
would still make sense.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
@ 2019-09-24 14:26 ` David Sterba
2019-09-24 15:34 ` Adam Borowski
0 siblings, 1 reply; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:26 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
I'll change the option to '-c' so we have the most common options as
lowercase letters.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
` (11 preceding siblings ...)
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
@ 2019-09-24 14:34 ` David Sterba
12 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 14:34 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 03, 2019 at 05:00:34PM +0200, Johannes Thumshirn wrote:
> Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
> prepared for an easy addition of new checksums, this patchset implements
> XXHASH64 as a second, fast but not cryptographically secure checksum hash.
>
> For changes since v2, please see the individual patches. Additionally a patch
> moving the CRC32C implementation from kernel-lib/ to crypto/ was added.
>
> For changes since v1, please see the individual patches. Additionally a unit
> test was added for regression testing this series.
>
>
> David Sterba (3):
> btrfs-progs: update checksumming api
> btrfs-progs: add xxhash sources
> btrfs-progs: add xxhash64 as checksum algorithm
>
> Johannes Thumshirn (9):
> btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
> btrfs-progs: cache csum_type in recover_control
> btrfs-progs: add checksum type to checksumming functions
> btrfs-progs: don't assume checksums are always 4 bytes
> btrfs-progs: pass checksum type to
> btrfs_csum_data()/btrfs_csum_final()
> btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
> btrfs-progs: add option for checksum type to mkfs
> btrfs-progs: move crc32c implementation to crypto/
> btrfs-progs: add test-case for mkfs with xxhash64
1-9 now in devel, please split 10 and see the comments for 12. You can
also update the documentation, I think the user interface is not going
to change significantly.
For first release we can use the builtin xxhash but as there's a
standalone library we'll need to add support for configure-time
selection. This will be even more necessary once SHA256 is added.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-24 14:26 ` David Sterba
@ 2019-09-24 15:34 ` Adam Borowski
2019-09-24 15:52 ` David Sterba
0 siblings, 1 reply; 23+ messages in thread
From: Adam Borowski @ 2019-09-24 15:34 UTC (permalink / raw)
To: dsterba, Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote:
> On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> > Add an option to mkfs to specify which checksum algorithm will be used for
> > the filesystem.
> >
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>
> I'll change the option to '-c' so we have the most common options as
> lowercase letters.
-c is used for compression elsewhere, I'd rather avoid this confusion.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs
2019-09-24 15:34 ` Adam Borowski
@ 2019-09-24 15:52 ` David Sterba
0 siblings, 0 replies; 23+ messages in thread
From: David Sterba @ 2019-09-24 15:52 UTC (permalink / raw)
To: Adam Borowski
Cc: dsterba, Johannes Thumshirn, David Sterba, Linux BTRFS Mailinglist
On Tue, Sep 24, 2019 at 05:34:11PM +0200, Adam Borowski wrote:
> On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote:
> > On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote:
> > > Add an option to mkfs to specify which checksum algorithm will be used for
> > > the filesystem.
> > >
> > > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> >
> > I'll change the option to '-c' so we have the most common options as
> > lowercase letters.
>
> -c is used for compression elsewhere, I'd rather avoid this confusion.
Fair point.
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2019-09-24 15:51 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-03 15:00 [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Johannes Thumshirn
2019-09-24 12:56 ` David Sterba
2019-09-24 13:06 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 07/12] btrfs-progs: update checksumming api Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-24 15:34 ` Adam Borowski
2019-09-24 15:52 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 09/12] btrfs-progs: add xxhash sources Johannes Thumshirn
2019-09-03 15:00 ` [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Johannes Thumshirn
2019-09-04 8:31 ` Nikolay Borisov
2019-09-24 14:23 ` David Sterba
2019-09-03 15:00 ` [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Johannes Thumshirn
2019-09-04 8:32 ` Nikolay Borisov
2019-09-03 15:00 ` [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Johannes Thumshirn
2019-09-24 14:26 ` David Sterba
2019-09-24 14:34 ` [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums David Sterba
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.