* [RFC 0/7] 64-bit data integrity field support @ 2022-01-24 16:01 Keith Busch 2022-01-24 16:01 ` [RFC 1/7] block: support pi with extended metadata Keith Busch ` (7 more replies) 0 siblings, 8 replies; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The NVM Express protocol added enhancements to the data integrity field formats beyond the T10 defined protection information. A detailed description of the new formats can be found in the NVMe's NVM Command Set Specification, section 5.2, available at: https://nvmexpress.org/wp-content/uploads/NVM-Command-Set-Specification-1.0b-2021.12.18-Ratified.pdf This series implements one possible new format: the CRC64 guard with 48-bit reference tags. This does not add support for the variable "storage tag" field. The NVMe CRC64 parameters (from Rocksoft) were not implemented in the kernel, so a software implementation is included in this series based on the generated table. This series does not include any possible hardware excelleration (ex: x86's pclmulqdq), so it's not very high performant right now. Keith Busch (7): block: support pi with extended metadata nvme: allow integrity on extended metadata formats lib: add rocksoft model crc64 lib: add crc64 tests asm-generic: introduce be48 unaligned accessors block: add pi for nvme enhanced integrity nvme: add support for enhanced metadata block/Kconfig | 1 + block/bio-integrity.c | 1 + block/t10-pi.c | 198 +++++++++++++++++++++++++++++++- drivers/nvme/host/core.c | 167 ++++++++++++++++++++++----- drivers/nvme/host/nvme.h | 1 + include/asm-generic/unaligned.h | 26 +++++ include/linux/blk-integrity.h | 1 + include/linux/crc64.h | 2 + include/linux/nvme.h | 53 ++++++++- include/linux/t10-pi.h | 20 ++++ lib/Kconfig.debug | 3 + lib/Makefile | 1 + lib/crc64.c | 79 +++++++++++++ lib/gen_crc64table.c | 33 ++++-- lib/test_crc64.c | 68 +++++++++++ 15 files changed, 608 insertions(+), 46 deletions(-) create mode 100644 lib/test_crc64.c -- 2.25.4 ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC 1/7] block: support pi with extended metadata 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-24 16:01 ` [RFC 2/7] nvme: allow integrity on extended metadata formats Keith Busch ` (6 subsequent siblings) 7 siblings, 0 replies; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The nvme spec allows protection information formats with metadata extending beyond the pi field. Use the actual size of the metadata field for incrementing the protection buffer. Cc: "Martin K. Petersen" <martin.petersen@oracle.com> Signed-off-by: Keith Busch <kbusch@kernel.org> --- block/bio-integrity.c | 1 + block/t10-pi.c | 4 ++-- include/linux/blk-integrity.h | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/block/bio-integrity.c b/block/bio-integrity.c index d25114715459..e40b1f965960 100644 --- a/block/bio-integrity.c +++ b/block/bio-integrity.c @@ -165,6 +165,7 @@ static blk_status_t bio_integrity_process(struct bio *bio, iter.disk_name = bio->bi_bdev->bd_disk->disk_name; iter.interval = 1 << bi->interval_exp; + iter.tuple_size = bi->tuple_size; iter.seed = proc_iter->bi_sector; iter.prot_buf = bvec_virt(bip->bip_vec); diff --git a/block/t10-pi.c b/block/t10-pi.c index 25a52a2a09a8..758a76518854 100644 --- a/block/t10-pi.c +++ b/block/t10-pi.c @@ -44,7 +44,7 @@ static blk_status_t t10_pi_generate(struct blk_integrity_iter *iter, pi->ref_tag = 0; iter->data_buf += iter->interval; - iter->prot_buf += sizeof(struct t10_pi_tuple); + iter->prot_buf += iter->tuple_size; iter->seed++; } @@ -93,7 +93,7 @@ static blk_status_t t10_pi_verify(struct blk_integrity_iter *iter, next: iter->data_buf += iter->interval; - iter->prot_buf += sizeof(struct t10_pi_tuple); + iter->prot_buf += iter->tuple_size; iter->seed++; } diff --git a/include/linux/blk-integrity.h b/include/linux/blk-integrity.h index 8a038ea0717e..378b2459efe2 100644 --- a/include/linux/blk-integrity.h +++ b/include/linux/blk-integrity.h @@ -19,6 +19,7 @@ struct blk_integrity_iter { sector_t seed; unsigned int data_size; unsigned short interval; + unsigned char tuple_size; const char *disk_name; }; -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC 2/7] nvme: allow integrity on extended metadata formats 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch 2022-01-24 16:01 ` [RFC 1/7] block: support pi with extended metadata Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-24 16:01 ` [RFC 3/7] lib: add rocksoft model crc64 Keith Busch ` (5 subsequent siblings) 7 siblings, 0 replies; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The block integrity subsystem knows how to construct protection information buffers with metadata beyond the protection information fields. Remove the driver restriction. Note, this can only work if the PI field appears first in the metadata, as the integrity subsystem doesn't update guard tags on preceding metadata. Signed-off-by: Keith Busch <kbusch@kernel.org> --- drivers/nvme/host/core.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 5e0bfda04bd7..b3eabf6a08b9 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1726,12 +1726,9 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id) { struct nvme_ctrl *ctrl = ns->ctrl; - /* - * The PI implementation requires the metadata size to be equal to the - * t10 pi tuple size. - */ ns->ms = le16_to_cpu(id->lbaf[id->flbas & NVME_NS_FLBAS_LBA_MASK].ms); - if (ns->ms == sizeof(struct t10_pi_tuple)) + if (id->dps & NVME_NS_DPS_PI_FIRST || + ns->ms == sizeof(struct t10_pi_tuple)) ns->pi_type = id->dps & NVME_NS_DPS_PI_MASK; else ns->pi_type = 0; -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC 3/7] lib: add rocksoft model crc64 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch 2022-01-24 16:01 ` [RFC 1/7] block: support pi with extended metadata Keith Busch 2022-01-24 16:01 ` [RFC 2/7] nvme: allow integrity on extended metadata formats Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-25 5:10 ` Eric Biggers 2022-01-24 16:01 ` [RFC 4/7] lib: add crc64 tests Keith Busch ` (4 subsequent siblings) 7 siblings, 1 reply; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The NVM Express specification extended data integrity fields to 64 bits using the Rocksoft^TM parameters. Add the poly (0xad93d23594c93659) to the crc64 table generation, and provide a library routine implementing the algorithm. Since this model reflects inputs and outputs, a helper table and function are added to reverse bits of 8 and 64 bit values. Cc: Coly Li <colyli@suse.de> Signed-off-by: Keith Busch <kbusch@kernel.org> --- include/linux/crc64.h | 2 ++ lib/crc64.c | 79 +++++++++++++++++++++++++++++++++++++++++++ lib/gen_crc64table.c | 33 ++++++++++++------ 3 files changed, 103 insertions(+), 11 deletions(-) diff --git a/include/linux/crc64.h b/include/linux/crc64.h index c756e65a1b58..9f2f20216503 100644 --- a/include/linux/crc64.h +++ b/include/linux/crc64.h @@ -8,4 +8,6 @@ #include <linux/types.h> u64 __pure crc64_be(u64 crc, const void *p, size_t len); +u64 __pure crc64_rocksoft(u64 crc, const void *p, size_t len); + #endif /* _LINUX_CRC64_H */ diff --git a/lib/crc64.c b/lib/crc64.c index 9f852a89ee2a..963283d1e682 100644 --- a/lib/crc64.c +++ b/lib/crc64.c @@ -22,6 +22,13 @@ * x^24 + x^23 + x^22 + x^21 + x^19 + x^17 + x^13 + x^12 + x^10 + x^9 + * x^7 + x^4 + x + 1 * + * crc64rocksoft[256] table is from the Rocksoft specification polynomial + * defined as, + * + * x^64 + x^63 + x^61 + x^59 + x^58 + x^56 + x^55 + x^52 + x^49 + x^48 + x^47 + + * x^46 + x^44 + x^41 + x^37 + x^36 + x^34 + x^32 + x^31 + x^28 + x^26 + x^23 + + * x^22 + x^19 + x^16 + x^13 + x^12 + x^10 + x^9 + x^6 + x^4 + x^3 + 1 + * * Copyright 2018 SUSE Linux. * Author: Coly Li <colyli@suse.de> */ @@ -55,3 +62,75 @@ u64 __pure crc64_be(u64 crc, const void *p, size_t len) return crc; } EXPORT_SYMBOL_GPL(crc64_be); + +static u64 bit_reverse(u64 val) +{ + unsigned long *v, *r; + u64 new_val = 0; + int i; + + v = (unsigned long *)&val; + r = (unsigned long *)&new_val; + + for_each_set_bit(i, v, 64) + set_bit(63 - i, r); + + return new_val; +} + +/* + * quick lookup table with values reversing the bits of the index + */ +static const u8 reverse[256] = { + 0x00, 0x80, 0x40, 0xC0, 0x20, 0xA0, 0x60, 0xE0, 0x10, 0x90, 0x50, 0xD0, + 0x30, 0xB0, 0x70, 0xF0, 0x08, 0x88, 0x48, 0xC8, 0x28, 0xA8, 0x68, 0xE8, + 0x18, 0x98, 0x58, 0xD8, 0x38, 0xB8, 0x78, 0xF8, 0x04, 0x84, 0x44, 0xC4, + 0x24, 0xA4, 0x64, 0xE4, 0x14, 0x94, 0x54, 0xD4, 0x34, 0xB4, 0x74, 0xF4, + 0x0C, 0x8C, 0x4C, 0xCC, 0x2C, 0xAC, 0x6C, 0xEC, 0x1C, 0x9C, 0x5C, 0xDC, + 0x3C, 0xBC, 0x7C, 0xFC, 0x02, 0x82, 0x42, 0xC2, 0x22, 0xA2, 0x62, 0xE2, + 0x12, 0x92, 0x52, 0xD2, 0x32, 0xB2, 0x72, 0xF2, 0x0A, 0x8A, 0x4A, 0xCA, + 0x2A, 0xAA, 0x6A, 0xEA, 0x1A, 0x9A, 0x5A, 0xDA, 0x3A, 0xBA, 0x7A, 0xFA, + 0x06, 0x86, 0x46, 0xC6, 0x26, 0xA6, 0x66, 0xE6, 0x16, 0x96, 0x56, 0xD6, + 0x36, 0xB6, 0x76, 0xF6, 0x0E, 0x8E, 0x4E, 0xCE, 0x2E, 0xAE, 0x6E, 0xEE, + 0x1E, 0x9E, 0x5E, 0xDE, 0x3E, 0xBE, 0x7E, 0xFE, 0x01, 0x81, 0x41, 0xC1, + 0x21, 0xA1, 0x61, 0xE1, 0x11, 0x91, 0x51, 0xD1, 0x31, 0xB1, 0x71, 0xF1, + 0x09, 0x89, 0x49, 0xC9, 0x29, 0xA9, 0x69, 0xE9, 0x19, 0x99, 0x59, 0xD9, + 0x39, 0xB9, 0x79, 0xF9, 0x05, 0x85, 0x45, 0xC5, 0x25, 0xA5, 0x65, 0xE5, + 0x15, 0x95, 0x55, 0xD5, 0x35, 0xB5, 0x75, 0xF5, 0x0D, 0x8D, 0x4D, 0xCD, + 0x2D, 0xAD, 0x6D, 0xED, 0x1D, 0x9D, 0x5D, 0xDD, 0x3D, 0xBD, 0x7D, 0xFD, + 0x03, 0x83, 0x43, 0xC3, 0x23, 0xA3, 0x63, 0xE3, 0x13, 0x93, 0x53, 0xD3, + 0x33, 0xB3, 0x73, 0xF3, 0x0B, 0x8B, 0x4B, 0xCB, 0x2B, 0xAB, 0x6B, 0xEB, + 0x1B, 0x9B, 0x5B, 0xDB, 0x3B, 0xBB, 0x7B, 0xFB, 0x07, 0x87, 0x47, 0xC7, + 0x27, 0xA7, 0x67, 0xE7, 0x17, 0x97, 0x57, 0xD7, 0x37, 0xB7, 0x77, 0xF7, + 0x0F, 0x8F, 0x4F, 0xCF, 0x2F, 0xAF, 0x6F, 0xEF, 0x1F, 0x9F, 0x5F, 0xDF, + 0x3F, 0xBF, 0x7F, 0xFF +}; + +/** + * crc64_rocksoft - Calculate bitwise Rocksoft CRC64 + * @crc: seed value for computation. (u64)~0 for a new CRC calculation, or the + * previous crc64 value if computing incrementally. + * @p: pointer to buffer over which CRC64 is run + * @len: length of buffer @p + * + * The Rocksoft 64-bit CRC model parameters: + * Initial value: 0xFFFFFFFFFFFFFFFF + * Reflected Input: True + * Reflected Output: True + * Xor Final: 0xFFFFFFFFFFFFFFFF + */ +u64 __pure crc64_rocksoft(u64 crc, const void *p, size_t len) +{ + size_t i, t; + + const unsigned char *_p = p; + + for (i = 0; i < len; i++) { + u8 tmp = reverse[*_p++]; + t = ((crc >> 56) ^ tmp) & 0xFF; + crc = crc64rocksofttable[t] ^ (crc << 8); + } + + return bit_reverse(crc) ^ (u64)~0; +} +EXPORT_SYMBOL_GPL(crc64_rocksoft); diff --git a/lib/gen_crc64table.c b/lib/gen_crc64table.c index 094b43aef8db..f1fceddb78ff 100644 --- a/lib/gen_crc64table.c +++ b/lib/gen_crc64table.c @@ -17,10 +17,12 @@ #include <stdio.h> #define CRC64_ECMA182_POLY 0x42F0E1EBA9EA3693ULL +#define CRC64_ROCKSOFT_POLY 0xAD93D23594C93659ULL static uint64_t crc64_table[256] = {0}; +static uint64_t crc64_rocksoft_table[256] = {0}; -static void generate_crc64_table(void) +static void generate_crc64_table(uint64_t table[256], uint64_t poly) { uint64_t i, j, c, crc; @@ -30,26 +32,22 @@ static void generate_crc64_table(void) for (j = 0; j < 8; j++) { if ((crc ^ c) & 0x8000000000000000ULL) - crc = (crc << 1) ^ CRC64_ECMA182_POLY; + crc = (crc << 1) ^ poly; else crc <<= 1; c <<= 1; } - crc64_table[i] = crc; + table[i] = crc; } } -static void print_crc64_table(void) +static void output_table(uint64_t table[256]) { int i; - printf("/* this file is generated - do not edit */\n\n"); - printf("#include <linux/types.h>\n"); - printf("#include <linux/cache.h>\n\n"); - printf("static const u64 ____cacheline_aligned crc64table[256] = {\n"); for (i = 0; i < 256; i++) { - printf("\t0x%016" PRIx64 "ULL", crc64_table[i]); + printf("\t0x%016" PRIx64 "ULL", table[i]); if (i & 0x1) printf(",\n"); else @@ -58,9 +56,22 @@ static void print_crc64_table(void) printf("};\n"); } +static void print_crc64_tables(void) +{ + printf("/* this file is generated - do not edit */\n\n"); + printf("#include <linux/types.h>\n"); + printf("#include <linux/cache.h>\n\n"); + printf("static const u64 ____cacheline_aligned crc64table[256] = {\n"); + output_table(crc64_table); + + printf("\nstatic const u64 ____cacheline_aligned crc64rocksofttable[256] = {\n"); + output_table(crc64_rocksoft_table); +} + int main(int argc, char *argv[]) { - generate_crc64_table(); - print_crc64_table(); + generate_crc64_table(crc64_table, CRC64_ECMA182_POLY); + generate_crc64_table(crc64_rocksoft_table, CRC64_ROCKSOFT_POLY); + print_crc64_tables(); return 0; } -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [RFC 3/7] lib: add rocksoft model crc64 2022-01-24 16:01 ` [RFC 3/7] lib: add rocksoft model crc64 Keith Busch @ 2022-01-25 5:10 ` Eric Biggers 2022-01-25 15:39 ` Keith Busch 0 siblings, 1 reply; 15+ messages in thread From: Eric Biggers @ 2022-01-25 5:10 UTC (permalink / raw) To: Keith Busch Cc: linux-nvme, linux-kernel, linux-block, axboe, hch, martin.petersen, colyli, arnd On Mon, Jan 24, 2022 at 08:01:03AM -0800, Keith Busch wrote: > Since this model reflects inputs and outputs, a helper table and > function are added to reverse bits of 8 and 64 bit values. That's a silly way to do a bit-reflected CRC. The proper way to do it is to reflect the bytes too, so that the bits and bytes are ordered consistently, so explicitly reflecting the bits isn't needed. Most CRC-32's are bit-reflected, and they are usually implemented this way. E.g., see crc32_le() in lib/crc32.c. Here's a Python script that shows that the Rocksoft CRC-64 can be computed without explicitly reversing the bits. It passes the tests from your patch 4: COEFFS = [63, 61, 59, 58, 56, 55, 52, 49, 48, 47, 46, 44, 41, 37, 36, 34, 32, 31, 28, 26, 23, 22, 19, 16, 13, 12, 10, 9, 6, 4, 3, 0] POLY = sum(1 << (63 - coeff) for coeff in COEFFS) # Generate the table. table = [0] * 256 for i in range(256): crc = 0 byte = i for j in range(8): if ((crc ^ (byte >> j)) & 1) == 1: crc = (crc >> 1) ^ POLY else: crc = crc >> 1 table[i] = crc # Compute the CRC-64 one byte at a time using the table. def crc64_rocksoft(data): crc = 0xffffffffffffffff for byte in data: crc = (crc >> 8) ^ table[(crc & 0xff) ^ byte] return crc ^ 0xffffffffffffffff # Tests assert crc64_rocksoft(bytearray([0] * 4096)) == 0x6482D367EB22B64E assert crc64_rocksoft(bytearray([255] * 4096)) == 0xC0DDBA7302ECA3AC assert crc64_rocksoft(bytearray([i % 256 for i in range(4096)])) == 0x3E729F5F6750449C assert crc64_rocksoft(bytearray([(255-i) % 256 for i in range(4096)])) == 0x9A2DF64B8E9E517E # Print the table. print(f'#define CRC64_ROCKSOFT_POLY 0x{POLY:016x}ULL') print('') print('static const u64 crc64_rocksoft_tab[] = {') for i in range(0, 256, 2): print('\t', end='') for j in range(i, i + 2): print(f'0x{table[j]:016x}ULL,', end='') if j != 1: print(' ', end='') print('') print('};') ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFC 3/7] lib: add rocksoft model crc64 2022-01-25 5:10 ` Eric Biggers @ 2022-01-25 15:39 ` Keith Busch 0 siblings, 0 replies; 15+ messages in thread From: Keith Busch @ 2022-01-25 15:39 UTC (permalink / raw) To: Eric Biggers Cc: linux-nvme, linux-kernel, linux-block, axboe, hch, martin.petersen, colyli, arnd On Mon, Jan 24, 2022 at 09:10:38PM -0800, Eric Biggers wrote: > On Mon, Jan 24, 2022 at 08:01:03AM -0800, Keith Busch wrote: > > Since this model reflects inputs and outputs, a helper table and > > function are added to reverse bits of 8 and 64 bit values. > > That's a silly way to do a bit-reflected CRC. The proper way to do it is to > reflect the bytes too, so that the bits and bytes are ordered consistently, so > explicitly reflecting the bits isn't needed. Most CRC-32's are bit-reflected, > and they are usually implemented this way. E.g., see crc32_le() in lib/crc32.c. > > Here's a Python script that shows that the Rocksoft CRC-64 can be computed > without explicitly reversing the bits. It passes the tests from your patch 4: Thanks for the tip! I'll use that for the next version. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC 4/7] lib: add crc64 tests 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch ` (2 preceding siblings ...) 2022-01-24 16:01 ` [RFC 3/7] lib: add rocksoft model crc64 Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-25 14:12 ` kernel test robot 2022-01-24 16:01 ` [RFC 5/7] asm-generic: introduce be48 unaligned accessors Keith Busch ` (3 subsequent siblings) 7 siblings, 1 reply; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch Provide a module to test the rocksoft crc64 calculations with well known inputs and exepected values. Signed-off-by: Keith Busch <kbusch@kernel.org> --- lib/Kconfig.debug | 3 +++ lib/Makefile | 1 + lib/test_crc64.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+) create mode 100644 lib/test_crc64.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index c77fe36bb3d8..97711884b270 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2212,6 +2212,9 @@ config TEST_UUID config TEST_XARRAY tristate "Test the XArray code at runtime" +config TEST_CRC64 + tristate "Test the crc64 code at runtime" + config TEST_OVERFLOW tristate "Test check_*_overflow() functions at runtime" diff --git a/lib/Makefile b/lib/Makefile index b213a7bbf3fd..423ef5ac3427 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -102,6 +102,7 @@ obj-$(CONFIG_TEST_HMM) += test_hmm.o obj-$(CONFIG_TEST_FREE_PAGES) += test_free_pages.o obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o obj-$(CONFIG_TEST_REF_TRACKER) += test_ref_tracker.o +obj-$(CONFIG_TEST_CRC64) += test_crc64.o # # CFLAGS for compiling floating point code inside the kernel. x86/Makefile turns # off the generation of FPU/SSE* instructions for kernel proper but FPU_FLAGS diff --git a/lib/test_crc64.c b/lib/test_crc64.c new file mode 100644 index 000000000000..283fef8f110e --- /dev/null +++ b/lib/test_crc64.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Tests were selected from NVM Express NVM Command Set Specification 1.0a, + * section 5.2.1.3.5 "64b CRC Test Cases" available here: + * + * https://nvmexpress.org/wp-content/uploads/NVMe-NVM-Command-Set-Specification-1.0a-2021.07.26-Ratified.pdf + * + * Copyright 2022 Keith Busch <kbusch@kernel.org> + */ + +#include <linux/crc64.h> +#include <linux/module.h> + +static unsigned int tests_passed; +static unsigned int tests_run; + +#define ALL_ZEROS 0x6482D367EB22B64EULL +#define ALL_FFS 0xC0DDBA7302ECA3ACULL +#define INC 0x3E729F5F6750449CULL +#define DEC 0x9A2DF64B8E9E517EULL + +static u8 buffer[4096]; + +#define CRC_CHECK(c, v) do { \ + tests_run++; \ + if (c != v) \ + printk("BUG at %s:%d expected:%llx got:%llx\n", \ + __func__, __LINE__, v, c); \ + else \ + tests_passed++; \ +} while (0) + + +static int crc_tests(void) +{ + __u64 crc; + int i; + + memset(buffer, 0, sizeof(buffer)); + crc = crc64_rocksoft(~0ULL, buffer, 4096); + CRC_CHECK(crc, ALL_ZEROS); + + memset(buffer, 0xff, sizeof(buffer)); + crc = crc64_rocksoft(~0ULL, buffer, 4096); + CRC_CHECK(crc, ALL_FFS); + + for (i = 0; i < 4096; i++) + buffer[i] = i & 0xff; + crc = crc64_rocksoft(~0ULL, buffer, 4096); + CRC_CHECK(crc, INC); + + for (i = 0; i < 4096; i++) + buffer[i] = 0xff - (i & 0xff); + crc = crc64_rocksoft(~0ULL, buffer, 4096); + CRC_CHECK(crc, DEC); + + printk("CRC64: %u of %u tests passed\n", tests_passed, tests_run); + return (tests_run == tests_passed) ? 0 : -EINVAL; +} + +static void crc_exit(void) +{ +} + +module_init(crc_tests); +module_exit(crc_exit); +MODULE_AUTHOR("Keith Busch <kbusch@kernel.org>"); +MODULE_LICENSE("GPL"); -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [RFC 4/7] lib: add crc64 tests 2022-01-24 16:01 ` [RFC 4/7] lib: add crc64 tests Keith Busch @ 2022-01-25 14:12 ` kernel test robot 0 siblings, 0 replies; 15+ messages in thread From: kernel test robot @ 2022-01-25 14:12 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 2596 bytes --] Hi Keith, [FYI, it's a private test report for your RFC patch.] [auto build test ERROR on axboe-block/for-next] [also build test ERROR on linux/master linus/master v5.17-rc1 next-20220125] [cannot apply to arnd-asm-generic/master linux-nvme/for-next] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Keith-Busch/64-bit-data-integrity-field-support/20220125-000232 base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next config: nios2-randconfig-r015-20220125 (https://download.01.org/0day-ci/archive/20220125/202201252238.BlrXAZDf-lkp(a)intel.com/config) compiler: nios2-linux-gcc (GCC) 11.2.0 reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # https://github.com/0day-ci/linux/commit/d45974b2937922cef5c20b3d5589364bc2d6cd8d git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Keith-Busch/64-bit-data-integrity-field-support/20220125-000232 git checkout d45974b2937922cef5c20b3d5589364bc2d6cd8d # save the config file to linux build tree mkdir build_dir COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=nios2 SHELL=/bin/bash If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All errors (new ones prefixed by >>): nios2-linux-ld: lib/test_crc64.o: in function `crc_tests': >> test_crc64.c:(.text+0x78): undefined reference to `crc64_rocksoft' test_crc64.c:(.text+0x78): relocation truncated to fit: R_NIOS2_CALL26 against `crc64_rocksoft' >> nios2-linux-ld: test_crc64.c:(.text+0xe4): undefined reference to `crc64_rocksoft' test_crc64.c:(.text+0xe4): relocation truncated to fit: R_NIOS2_CALL26 against `crc64_rocksoft' nios2-linux-ld: test_crc64.c:(.text+0x158): undefined reference to `crc64_rocksoft' test_crc64.c:(.text+0x158): relocation truncated to fit: R_NIOS2_CALL26 against `crc64_rocksoft' nios2-linux-ld: test_crc64.c:(.text+0x1c8): undefined reference to `crc64_rocksoft' test_crc64.c:(.text+0x1c8): relocation truncated to fit: R_NIOS2_CALL26 against `crc64_rocksoft' --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC 5/7] asm-generic: introduce be48 unaligned accessors 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch ` (3 preceding siblings ...) 2022-01-24 16:01 ` [RFC 4/7] lib: add crc64 tests Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-24 17:38 ` Arnd Bergmann 2022-01-24 16:01 ` [RFC 6/7] block: add pi for nvme enhanced integrity Keith Busch ` (2 subsequent siblings) 7 siblings, 1 reply; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The NVMe protocol extended data integrity fields with unaligned 48-bit reference tags. Provide some helper accessors in preparation for these. Cc: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Keith Busch <kbusch@kernel.org> --- include/asm-generic/unaligned.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/include/asm-generic/unaligned.h b/include/asm-generic/unaligned.h index 1c4242416c9f..8fc637379899 100644 --- a/include/asm-generic/unaligned.h +++ b/include/asm-generic/unaligned.h @@ -126,4 +126,30 @@ static inline void put_unaligned_le24(const u32 val, void *p) __put_unaligned_le24(val, p); } +static inline void __put_unaligned_be48(const u64 val, __u8 *p) +{ + *p++ = val >> 40; + *p++ = val >> 32; + *p++ = val >> 24; + *p++ = val >> 16; + *p++ = val >> 8; + *p++ = val; +} + +static inline void put_unaligned_be48(const u64 val, void *p) +{ + __put_unaligned_be48(val, p); +} + +static inline u64 __get_unaligned_be48(const u8 *p) +{ + return (u64)p[0] << 40 | (u64)p[1] << 32 | p[2] << 24 | + p[3] << 16 | p[4] << 8 | p[5]; +} + +static inline u64 get_unaligned_be48(const void *p) +{ + return __get_unaligned_be48(p); +} + #endif /* __ASM_GENERIC_UNALIGNED_H */ -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [RFC 5/7] asm-generic: introduce be48 unaligned accessors 2022-01-24 16:01 ` [RFC 5/7] asm-generic: introduce be48 unaligned accessors Keith Busch @ 2022-01-24 17:38 ` Arnd Bergmann 0 siblings, 0 replies; 15+ messages in thread From: Arnd Bergmann @ 2022-01-24 17:38 UTC (permalink / raw) To: Keith Busch Cc: linux-nvme, Linux Kernel Mailing List, linux-block, Jens Axboe, Christoph Hellwig, Martin K. Petersen, Coly Li, Arnd Bergmann On Mon, Jan 24, 2022 at 5:01 PM Keith Busch <kbusch@kernel.org> wrote: > > The NVMe protocol extended data integrity fields with unaligned 48-bit > reference tags. Provide some helper accessors in preparation for these. > > Cc: Arnd Bergmann <arnd@arndb.de> > Signed-off-by: Keith Busch <kbusch@kernel.org> For asm-generic: Acked-by: Arnd Bergmann <arnd@arndb.de> I assume you are keeping this together with the rest of the series. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC 6/7] block: add pi for nvme enhanced integrity 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch ` (4 preceding siblings ...) 2022-01-24 16:01 ` [RFC 5/7] asm-generic: introduce be48 unaligned accessors Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-24 16:01 ` [RFC 7/7] nvme: add support for enhanced metadata Keith Busch 2022-01-26 14:38 ` [RFC 0/7] 64-bit data integrity field support Klaus Jensen 7 siblings, 0 replies; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch The NVMe specification defines larger data integrity formats beyond the t10 tuple. Add support for the specification defined CRC64 formats, assuming the reference tag does not need to be split with the "storage tag". Cc: "Martin K. Petersen" <martin.petersen@oracle.com> Signed-off-by: Keith Busch <kbusch@kernel.org> --- block/Kconfig | 1 + block/t10-pi.c | 194 +++++++++++++++++++++++++++++++++++++++++ include/linux/t10-pi.h | 20 +++++ 3 files changed, 215 insertions(+) diff --git a/block/Kconfig b/block/Kconfig index d5d4197b7ed2..78eca050dff7 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -63,6 +63,7 @@ config BLK_DEV_INTEGRITY_T10 tristate depends on BLK_DEV_INTEGRITY select CRC_T10DIF + select CRC64 config BLK_DEV_ZONED bool "Zoned block device support" diff --git a/block/t10-pi.c b/block/t10-pi.c index 758a76518854..7bfefe970bc5 100644 --- a/block/t10-pi.c +++ b/block/t10-pi.c @@ -7,8 +7,10 @@ #include <linux/t10-pi.h> #include <linux/blk-integrity.h> #include <linux/crc-t10dif.h> +#include <linux/crc64.h> #include <linux/module.h> #include <net/checksum.h> +#include <asm/unaligned.h> typedef __be16 (csum_fn) (void *, unsigned int); @@ -278,4 +280,196 @@ const struct blk_integrity_profile t10_pi_type3_ip = { }; EXPORT_SYMBOL(t10_pi_type3_ip); +static __be64 nvme_pi_crc64(void *data, unsigned int len) +{ + return cpu_to_be64(crc64_rocksoft(~0ULL, data, len)); +} + +static blk_status_t nvme_crc64_generate(struct blk_integrity_iter *iter, + enum t10_dif_type type) +{ + unsigned int i; + + for (i = 0 ; i < iter->data_size ; i += iter->interval) { + struct nvme_crc64_pi_tuple *pi = iter->prot_buf; + + pi->guard_tag = nvme_pi_crc64(iter->data_buf, iter->interval); + pi->app_tag = 0; + + if (type == T10_PI_TYPE1_PROTECTION) + put_unaligned_be48(iter->seed, pi->ref_tag); + else + put_unaligned_be48(0ULL, pi->ref_tag); + + iter->data_buf += iter->interval; + iter->prot_buf += iter->tuple_size; + iter->seed++; + } + + return BLK_STS_OK; +} + +static bool nvme_crc64_ref_escape(u8 *ref_tag) +{ + static u8 ref_escape[6] = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff }; + + return memcmp(ref_tag, ref_escape, sizeof(ref_escape)) == 0; +} + +static blk_status_t nvme_crc64_verify(struct blk_integrity_iter *iter, + enum t10_dif_type type) +{ + unsigned int i; + + for (i = 0; i < iter->data_size; i += iter->interval) { + struct nvme_crc64_pi_tuple *pi = iter->prot_buf; + u64 ref, seed; + __be64 csum; + + if (type == T10_PI_TYPE1_PROTECTION) { + if (pi->app_tag == T10_PI_APP_ESCAPE) + goto next; + + ref = get_unaligned_be48(pi->ref_tag); + seed = iter->seed & 0xffffffffffffull; + if (ref != seed) { + pr_err("%s: ref tag error at location %llu (rcvd %llu)\n", + iter->disk_name, seed, ref); + return BLK_STS_PROTECTION; + } + } else if (type == T10_PI_TYPE3_PROTECTION) { + if (pi->app_tag == T10_PI_APP_ESCAPE && + nvme_crc64_ref_escape(pi->ref_tag)) + goto next; + } + + csum = nvme_pi_crc64(iter->data_buf, iter->interval); + if (pi->guard_tag != csum) { + pr_err("%s: guard tag error at sector %llu " \ + "(rcvd %016llx, want %016llx)\n", + iter->disk_name, (unsigned long long)iter->seed, + be64_to_cpu(pi->guard_tag), be64_to_cpu(csum)); + return BLK_STS_PROTECTION; + } + +next: + iter->data_buf += iter->interval; + iter->prot_buf += iter->tuple_size; + iter->seed++; + } + + return BLK_STS_OK; +} + +static blk_status_t nvme_pi_type1_verify_crc(struct blk_integrity_iter *iter) +{ + return nvme_crc64_verify(iter, T10_PI_TYPE1_PROTECTION); +} + +static blk_status_t nvme_pi_type1_generate_crc(struct blk_integrity_iter *iter) +{ + return nvme_crc64_generate(iter, T10_PI_TYPE1_PROTECTION); +} + +static void nvme_pi_type1_prepare(struct request *rq) +{ + const int tuple_sz = rq->q->integrity.tuple_size; + u64 ref_tag = nvme_pi_extended_ref_tag(rq); + struct bio *bio; + + __rq_for_each_bio(bio, rq) { + struct bio_integrity_payload *bip = bio_integrity(bio); + u64 virt = bip_get_seed(bip) & 0xffffffffffffull; + struct bio_vec iv; + struct bvec_iter iter; + + /* Already remapped? */ + if (bip->bip_flags & BIP_MAPPED_INTEGRITY) + break; + + bip_for_each_vec(iv, bip, iter) { + unsigned int j; + void *p; + + p = bvec_kmap_local(&iv); + for (j = 0; j < iv.bv_len; j += tuple_sz) { + struct nvme_crc64_pi_tuple *pi = p; + u64 ref = get_unaligned_be48(pi->ref_tag); + + if (ref == virt) + put_unaligned_be48(ref_tag, pi->ref_tag); + virt++; + ref_tag++; + p += tuple_sz; + } + kunmap_local(p); + } + + bip->bip_flags |= BIP_MAPPED_INTEGRITY; + } +} + +static void nvme_pi_type1_complete(struct request *rq, unsigned int nr_bytes) +{ + unsigned intervals = nr_bytes >> rq->q->integrity.interval_exp; + const int tuple_sz = rq->q->integrity.tuple_size; + u64 ref_tag = nvme_pi_extended_ref_tag(rq); + struct bio *bio; + + __rq_for_each_bio(bio, rq) { + struct bio_integrity_payload *bip = bio_integrity(bio); + u64 virt = bip_get_seed(bip) & 0xffffffffffffull; + struct bio_vec iv; + struct bvec_iter iter; + + bip_for_each_vec(iv, bip, iter) { + unsigned int j; + void *p; + + p = bvec_kmap_local(&iv); + for (j = 0; j < iv.bv_len && intervals; j += tuple_sz) { + struct nvme_crc64_pi_tuple *pi = p; + u64 ref = get_unaligned_be48(pi->ref_tag); + + if (ref == ref_tag) + put_unaligned_be48(virt, pi->ref_tag); + virt++; + ref_tag++; + intervals--; + p += tuple_sz; + } + kunmap_local(p); + } + } +} + +static blk_status_t nvme_pi_type3_verify_crc(struct blk_integrity_iter *iter) +{ + return nvme_crc64_verify(iter, T10_PI_TYPE3_PROTECTION); +} + +static blk_status_t nvme_pi_type3_generate_crc(struct blk_integrity_iter *iter) +{ + return nvme_crc64_generate(iter, T10_PI_TYPE3_PROTECTION); +} + +const struct blk_integrity_profile nvme_pi_type1_crc64 = { + .name = "NVME-DIF-TYPE1-CRC64", + .generate_fn = nvme_pi_type1_generate_crc, + .verify_fn = nvme_pi_type1_verify_crc, + .prepare_fn = nvme_pi_type1_prepare, + .complete_fn = nvme_pi_type1_complete, +}; +EXPORT_SYMBOL(nvme_pi_type1_crc64); + +const struct blk_integrity_profile nvme_pi_type3_crc64 = { + .name = "NVME-DIF-TYPE3-CRC64", + .generate_fn = nvme_pi_type3_generate_crc, + .verify_fn = nvme_pi_type3_verify_crc, + .prepare_fn = t10_pi_type3_prepare, + .complete_fn = t10_pi_type3_complete, +}; +EXPORT_SYMBOL(nvme_pi_type3_crc64); + +MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL"); diff --git a/include/linux/t10-pi.h b/include/linux/t10-pi.h index c635c2e014e3..86bc8713ca09 100644 --- a/include/linux/t10-pi.h +++ b/include/linux/t10-pi.h @@ -53,4 +53,24 @@ extern const struct blk_integrity_profile t10_pi_type1_ip; extern const struct blk_integrity_profile t10_pi_type3_crc; extern const struct blk_integrity_profile t10_pi_type3_ip; +struct nvme_crc64_pi_tuple { + __be64 guard_tag; + __be16 app_tag; + __u8 ref_tag[6]; +}; + +static inline u64 nvme_pi_extended_ref_tag(struct request *rq) +{ + unsigned int shift = ilog2(queue_logical_block_size(rq->q)); + +#ifdef CONFIG_BLK_DEV_INTEGRITY + if (rq->q->integrity.interval_exp) + shift = rq->q->integrity.interval_exp; +#endif + return blk_rq_pos(rq) >> (shift - SECTOR_SHIFT) & 0xffffffffffffull; +} + +extern const struct blk_integrity_profile nvme_pi_type1_crc64; +extern const struct blk_integrity_profile nvme_pi_type3_crc64; + #endif -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC 7/7] nvme: add support for enhanced metadata 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch ` (5 preceding siblings ...) 2022-01-24 16:01 ` [RFC 6/7] block: add pi for nvme enhanced integrity Keith Busch @ 2022-01-24 16:01 ` Keith Busch 2022-01-24 22:23 ` kernel test robot 2022-01-26 14:38 ` [RFC 0/7] 64-bit data integrity field support Klaus Jensen 7 siblings, 1 reply; 15+ messages in thread From: Keith Busch @ 2022-01-24 16:01 UTC (permalink / raw) To: linux-nvme, linux-kernel, linux-block Cc: axboe, hch, martin.petersen, colyli, arnd, Keith Busch NVM Express ratified TP 4069 defines new protection information formats. Implement support for the CRC64 guard tags. Since the block layer doesn't support variable length reference tags, driver support for the Storage Tag space is not supported at this time. Signed-off-by: Keith Busch <kbusch@kernel.org> --- drivers/nvme/host/core.c | 164 +++++++++++++++++++++++++++++++++------ drivers/nvme/host/nvme.h | 1 + include/linux/nvme.h | 53 +++++++++++-- 3 files changed, 188 insertions(+), 30 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index b3eabf6a08b9..65534ed646e4 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -882,6 +882,30 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, return BLK_STS_OK; } +static inline void nvme_set_ref_tag(struct nvme_ns *ns, struct nvme_command *cmnd, + struct request *req) +{ + u32 upper, lower; + u64 ref48; + + /* both rw and write zeroes share the same reftag format */ + switch (ns->guard_type) { + case NVME_NVM_NS_16B_GUARD: + cmnd->rw.reftag = cpu_to_le32(t10_pi_ref_tag(req)); + break; + case NVME_NVM_NS_64B_GUARD: + ref48 = nvme_pi_extended_ref_tag(req); + lower = lower_32_bits(ref48); + upper = upper_32_bits(ref48); + + cmnd->rw.reftag = cpu_to_le32(lower); + cmnd->rw.cdw3 = cpu_to_le32(upper); + break; + default: + break; + } +} + static inline blk_status_t nvme_setup_write_zeroes(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -903,8 +927,7 @@ static inline blk_status_t nvme_setup_write_zeroes(struct nvme_ns *ns, switch (ns->pi_type) { case NVME_NS_DPS_PI_TYPE1: case NVME_NS_DPS_PI_TYPE2: - cmnd->write_zeroes.reftag = - cpu_to_le32(t10_pi_ref_tag(req)); + nvme_set_ref_tag(ns, cmnd, req); break; } } @@ -931,7 +954,8 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, cmnd->rw.opcode = op; cmnd->rw.flags = 0; cmnd->rw.nsid = cpu_to_le32(ns->head->ns_id); - cmnd->rw.rsvd2 = 0; + cmnd->rw.cdw2 = 0; + cmnd->rw.cdw3 = 0; cmnd->rw.metadata = 0; cmnd->rw.slba = cpu_to_le64(nvme_sect_to_lba(ns, blk_rq_pos(req))); cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1); @@ -965,7 +989,7 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, NVME_RW_PRINFO_PRCHK_REF; if (op == nvme_cmd_zone_append) control |= NVME_RW_APPEND_PIREMAP; - cmnd->rw.reftag = cpu_to_le32(t10_pi_ref_tag(req)); + nvme_set_ref_tag(ns, cmnd, req); break; } } @@ -1619,33 +1643,58 @@ int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo) } #ifdef CONFIG_BLK_DEV_INTEGRITY -static void nvme_init_integrity(struct gendisk *disk, u16 ms, u8 pi_type, +static void nvme_init_integrity(struct gendisk *disk, struct nvme_ns *ns, u32 max_integrity_segments) { struct blk_integrity integrity = { }; - switch (pi_type) { + switch (ns->pi_type) { case NVME_NS_DPS_PI_TYPE3: - integrity.profile = &t10_pi_type3_crc; - integrity.tag_size = sizeof(u16) + sizeof(u32); - integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + switch (ns->guard_type) { + case NVME_NVM_NS_16B_GUARD: + integrity.profile = &t10_pi_type3_crc; + integrity.tag_size = sizeof(u16) + sizeof(u32); + integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + break; + case NVME_NVM_NS_64B_GUARD: + integrity.profile = &nvme_pi_type1_crc64; + integrity.tag_size = sizeof(u16) + 6; + integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + break; + default: + integrity.profile = NULL; + break; + } break; case NVME_NS_DPS_PI_TYPE1: case NVME_NS_DPS_PI_TYPE2: - integrity.profile = &t10_pi_type1_crc; - integrity.tag_size = sizeof(u16); - integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + switch (ns->guard_type) { + case NVME_NVM_NS_16B_GUARD: + integrity.profile = &t10_pi_type1_crc; + integrity.tag_size = sizeof(u16); + integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + break; + case NVME_NVM_NS_64B_GUARD: + integrity.profile = &nvme_pi_type1_crc64; + integrity.tag_size = sizeof(u16); + integrity.flags |= BLK_INTEGRITY_DEVICE_CAPABLE; + break; + default: + integrity.profile = NULL; + break; + } break; default: integrity.profile = NULL; break; } - integrity.tuple_size = ms; + + integrity.tuple_size = ns->ms; blk_integrity_register(disk, &integrity); blk_queue_max_integrity_segments(disk->queue, max_integrity_segments); } #else -static void nvme_init_integrity(struct gendisk *disk, u16 ms, u8 pi_type, +static void nvme_init_integrity(struct gendisk *disk, struct nvme_ns *ns, u32 max_integrity_segments) { } @@ -1722,17 +1771,75 @@ static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns, return 0; } -static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id) +static int nvme_init_ms(struct nvme_ns *ns, struct nvme_id_ns *id) { + bool first = id->dps & NVME_NS_DPS_PI_FIRST; + unsigned lbaf = nvme_lbaf_index(id->flbas); struct nvme_ctrl *ctrl = ns->ctrl; + struct nvme_command c = { }; + struct nvme_id_ns_nvm *nvm; + u8 pi_size = 0; + int ret = 0; + u32 elbaf; + + ns->ms = le16_to_cpu(id->lbaf[lbaf].ms); + if (!(ctrl->ctratt & NVME_CTRL_ATTR_ELBAS)) { + pi_size = sizeof(struct t10_pi_tuple); + ns->guard_type = NVME_NVM_NS_16B_GUARD; + goto set_pi; + } + + nvm = kzalloc(sizeof(*nvm), GFP_KERNEL); + if (!nvm) + return -ENOMEM; - ns->ms = le16_to_cpu(id->lbaf[id->flbas & NVME_NS_FLBAS_LBA_MASK].ms); - if (id->dps & NVME_NS_DPS_PI_FIRST || - ns->ms == sizeof(struct t10_pi_tuple)) + c.identify.opcode = nvme_admin_identify; + c.identify.nsid = cpu_to_le32(ns->head->ns_id); + c.identify.cns = NVME_ID_CNS_CS_NS; + c.identify.csi = NVME_CSI_NVM; + + ret = nvme_submit_sync_cmd(ns->ctrl->admin_q, &c, nvm, sizeof(*nvm)); + if (ret) + goto free_data; + + elbaf = cpu_to_le32(nvm->elbaf[lbaf]); + + /* no support for storage tag formats right now */ + if (nvme_elbaf_sts(elbaf)) + goto free_data; + + ns->guard_type = nvme_elbaf_guard_type(elbaf); + switch (ns->guard_type) { + case NVME_NVM_NS_64B_GUARD: + pi_size = sizeof(struct nvme_crc64_pi_tuple); + break; + case NVME_NVM_NS_16B_GUARD: + pi_size = sizeof(struct t10_pi_tuple); + break; + default: + break; + } + +free_data: + kfree(nvm); +set_pi: + if (pi_size && (first || ns->ms == pi_size)) ns->pi_type = id->dps & NVME_NS_DPS_PI_MASK; else ns->pi_type = 0; + return ret; +} + +static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + int ret; + + ret = nvme_init_ms(ns, id); + if (ret) + return ret; + ns->features &= ~(NVME_NS_METADATA_SUPPORTED | NVME_NS_EXT_LBAS); if (!ns->ms || !(ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)) return 0; @@ -1850,7 +1957,7 @@ static void nvme_update_disk_info(struct gendisk *disk, if (ns->ms) { if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && (ns->features & NVME_NS_METADATA_SUPPORTED)) - nvme_init_integrity(disk, ns->ms, ns->pi_type, + nvme_init_integrity(disk, ns, ns->ctrl->max_integrity_segments); else if (!nvme_ns_has_pi(ns)) capacity = 0; @@ -1905,7 +2012,7 @@ static void nvme_set_chunk_sectors(struct nvme_ns *ns, struct nvme_id_ns *id) static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id) { - unsigned lbaf = id->flbas & NVME_NS_FLBAS_LBA_MASK; + unsigned lbaf = nvme_lbaf_index(id->flbas); int ret; blk_mq_freeze_queue(ns->disk->queue); @@ -2252,20 +2359,27 @@ static int nvme_configure_timestamp(struct nvme_ctrl *ctrl) return ret; } -static int nvme_configure_acre(struct nvme_ctrl *ctrl) +static int nvme_configure_host_options(struct nvme_ctrl *ctrl) { struct nvme_feat_host_behavior *host; + u8 acre = 0, lbafee = 0; int ret; /* Don't bother enabling the feature if retry delay is not reported */ - if (!ctrl->crdt[0]) + if (ctrl->crdt[0]) + acre = NVME_ENABLE_ACRE; + if (ctrl->ctratt & NVME_CTRL_ATTR_ELBAS) + lbafee = NVME_ENABLE_LBAFEE; + + if (!acre && !lbafee) return 0; host = kzalloc(sizeof(*host), GFP_KERNEL); if (!host) return 0; - host->acre = NVME_ENABLE_ACRE; + host->acre = acre; + host->lbafee = lbafee; ret = nvme_set_features(ctrl, NVME_FEAT_HOST_BEHAVIOR, 0, host, sizeof(*host), NULL); kfree(host); @@ -3104,7 +3218,7 @@ int nvme_init_ctrl_finish(struct nvme_ctrl *ctrl) if (ret < 0) return ret; - ret = nvme_configure_acre(ctrl); + ret = nvme_configure_host_options(ctrl); if (ret < 0) return ret; @@ -4725,12 +4839,14 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_id_ctrl) != NVME_IDENTIFY_DATA_SIZE); BUILD_BUG_ON(sizeof(struct nvme_id_ns) != NVME_IDENTIFY_DATA_SIZE); BUILD_BUG_ON(sizeof(struct nvme_id_ns_zns) != NVME_IDENTIFY_DATA_SIZE); + BUILD_BUG_ON(sizeof(struct nvme_id_ns_nvm) != NVME_IDENTIFY_DATA_SIZE); BUILD_BUG_ON(sizeof(struct nvme_id_ctrl_zns) != NVME_IDENTIFY_DATA_SIZE); BUILD_BUG_ON(sizeof(struct nvme_id_ctrl_nvm) != NVME_IDENTIFY_DATA_SIZE); BUILD_BUG_ON(sizeof(struct nvme_lba_range_type) != 64); BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512); BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64); BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_feat_host_behavior) != 512); } diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index a162f6c6da6e..175a2a5fbd73 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -455,6 +455,7 @@ struct nvme_ns { u16 sgs; u32 sws; u8 pi_type; + u8 guard_type; #ifdef CONFIG_BLK_DEV_ZONED u64 zsze; #endif diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 855dd9b3e84b..4342b7eed3e2 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -238,6 +238,7 @@ enum { enum nvme_ctrl_attr { NVME_CTRL_ATTR_HID_128_BIT = (1 << 0), NVME_CTRL_ATTR_TBKAS = (1 << 6), + NVME_CTRL_ATTR_ELBAS = (1 << 15), }; struct nvme_id_ctrl { @@ -391,8 +392,7 @@ struct nvme_id_ns { __le16 endgid; __u8 nguid[16]; __u8 eui64[8]; - struct nvme_lbaf lbaf[16]; - __u8 rsvd192[192]; + struct nvme_lbaf lbaf[64]; __u8 vs[3712]; }; @@ -410,8 +410,7 @@ struct nvme_id_ns_zns { __le32 rrl; __le32 frl; __u8 rsvd20[2796]; - struct nvme_zns_lbafe lbafe[16]; - __u8 rsvd3072[768]; + struct nvme_zns_lbafe lbafe[64]; __u8 vs[256]; }; @@ -420,6 +419,30 @@ struct nvme_id_ctrl_zns { __u8 rsvd1[4095]; }; +struct nvme_id_ns_nvm { + __le64 lbstm; + __u8 pic; + __u8 rsvd9[3]; + __le32 elbaf[64]; + __u8 rsvd268[3828]; +}; + +enum { + NVME_ID_NS_NVM_STS_MASK = 0x3f, + NVME_ID_NS_NVM_GUARD_SHIFT = 7, + NVME_ID_NS_NVM_GUARD_MASK = 0x3, +}; + +static inline __u8 nvme_elbaf_sts(__u32 elbaf) +{ + return elbaf & NVME_ID_NS_NVM_STS_MASK; +} + +static inline __u8 nvme_elbaf_guard_type(__u32 elbaf) +{ + return (elbaf >> NVME_ID_NS_NVM_GUARD_SHIFT) & NVME_ID_NS_NVM_GUARD_MASK; +} + struct nvme_id_ctrl_nvm { __u8 vsl; __u8 wzsl; @@ -470,6 +493,8 @@ enum { NVME_NS_FEAT_IO_OPT = 1 << 4, NVME_NS_ATTR_RO = 1 << 0, NVME_NS_FLBAS_LBA_MASK = 0xf, + NVME_NS_FLBAS_LBA_UMASK = 0x60, + NVME_NS_FLBAS_LBA_SHIFT = 1, NVME_NS_FLBAS_META_EXT = 0x10, NVME_NS_NMIC_SHARED = 1 << 0, NVME_LBAF_RP_BEST = 0, @@ -488,6 +513,18 @@ enum { NVME_NS_DPS_PI_TYPE3 = 3, }; +enum { + NVME_NVM_NS_16B_GUARD = 0, + NVME_NVM_NS_32B_GUARD = 1, + NVME_NVM_NS_64B_GUARD = 2, +}; + +static inline __u8 nvme_lbaf_index(__u8 flbas) +{ + return (flbas & NVME_NS_FLBAS_LBA_MASK) | + ((flbas & NVME_NS_FLBAS_LBA_UMASK) >> NVME_NS_FLBAS_LBA_SHIFT); +} + /* Identify Namespace Metadata Capabilities (MC): */ enum { NVME_MC_EXTENDED_LBA = (1 << 0), @@ -834,7 +871,8 @@ struct nvme_rw_command { __u8 flags; __u16 command_id; __le32 nsid; - __u64 rsvd2; + __le32 cdw2; + __le32 cdw3; __le64 metadata; union nvme_data_ptr dptr; __le64 slba; @@ -988,11 +1026,14 @@ enum { struct nvme_feat_host_behavior { __u8 acre; - __u8 resv1[511]; + __u8 etdas; + __u8 lbafee; + __u8 resv1[509]; }; enum { NVME_ENABLE_ACRE = 1, + NVME_ENABLE_LBAFEE = 1, }; /* Admin commands */ -- 2.25.4 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [RFC 7/7] nvme: add support for enhanced metadata 2022-01-24 16:01 ` [RFC 7/7] nvme: add support for enhanced metadata Keith Busch @ 2022-01-24 22:23 ` kernel test robot 0 siblings, 0 replies; 15+ messages in thread From: kernel test robot @ 2022-01-24 22:23 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 4039 bytes --] Hi Keith, [FYI, it's a private test report for your RFC patch.] [auto build test WARNING on axboe-block/for-next] [also build test WARNING on linux/master linus/master v5.17-rc1 next-20220124] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Keith-Busch/64-bit-data-integrity-field-support/20220125-000232 base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next config: x86_64-randconfig-s021-20220124 (https://download.01.org/0day-ci/archive/20220125/202201250552.LY0QkQNT-lkp(a)intel.com/config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.4-dirty # https://github.com/0day-ci/linux/commit/15e965a414da565fcf7d7aa958a9850bb714db17 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Keith-Busch/64-bit-data-integrity-field-support/20220125-000232 git checkout 15e965a414da565fcf7d7aa958a9850bb714db17 # save the config file to linux build tree mkdir build_dir make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=x86_64 SHELL=/bin/bash drivers/nvme/host/ If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> sparse warnings: (new ones prefixed by >>) >> drivers/nvme/host/core.c:1805:17: sparse: sparse: cast from restricted __le32 >> drivers/nvme/host/core.c:1805:15: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned int [usertype] elbaf @@ got restricted __le32 [usertype] @@ drivers/nvme/host/core.c:1805:15: sparse: expected unsigned int [usertype] elbaf drivers/nvme/host/core.c:1805:15: sparse: got restricted __le32 [usertype] vim +1805 drivers/nvme/host/core.c 1773 1774 static int nvme_init_ms(struct nvme_ns *ns, struct nvme_id_ns *id) 1775 { 1776 bool first = id->dps & NVME_NS_DPS_PI_FIRST; 1777 unsigned lbaf = nvme_lbaf_index(id->flbas); 1778 struct nvme_ctrl *ctrl = ns->ctrl; 1779 struct nvme_command c = { }; 1780 struct nvme_id_ns_nvm *nvm; 1781 u8 pi_size = 0; 1782 int ret = 0; 1783 u32 elbaf; 1784 1785 ns->ms = le16_to_cpu(id->lbaf[lbaf].ms); 1786 if (!(ctrl->ctratt & NVME_CTRL_ATTR_ELBAS)) { 1787 pi_size = sizeof(struct t10_pi_tuple); 1788 ns->guard_type = NVME_NVM_NS_16B_GUARD; 1789 goto set_pi; 1790 } 1791 1792 nvm = kzalloc(sizeof(*nvm), GFP_KERNEL); 1793 if (!nvm) 1794 return -ENOMEM; 1795 1796 c.identify.opcode = nvme_admin_identify; 1797 c.identify.nsid = cpu_to_le32(ns->head->ns_id); 1798 c.identify.cns = NVME_ID_CNS_CS_NS; 1799 c.identify.csi = NVME_CSI_NVM; 1800 1801 ret = nvme_submit_sync_cmd(ns->ctrl->admin_q, &c, nvm, sizeof(*nvm)); 1802 if (ret) 1803 goto free_data; 1804 > 1805 elbaf = cpu_to_le32(nvm->elbaf[lbaf]); 1806 1807 /* no support for storage tag formats right now */ 1808 if (nvme_elbaf_sts(elbaf)) 1809 goto free_data; 1810 1811 ns->guard_type = nvme_elbaf_guard_type(elbaf); 1812 switch (ns->guard_type) { 1813 case NVME_NVM_NS_64B_GUARD: 1814 pi_size = sizeof(struct nvme_crc64_pi_tuple); 1815 break; 1816 case NVME_NVM_NS_16B_GUARD: 1817 pi_size = sizeof(struct t10_pi_tuple); 1818 break; 1819 default: 1820 break; 1821 } 1822 1823 free_data: 1824 kfree(nvm); 1825 set_pi: 1826 if (pi_size && (first || ns->ms == pi_size)) 1827 ns->pi_type = id->dps & NVME_NS_DPS_PI_MASK; 1828 else 1829 ns->pi_type = 0; 1830 1831 return ret; 1832 } 1833 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFC 0/7] 64-bit data integrity field support 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch ` (6 preceding siblings ...) 2022-01-24 16:01 ` [RFC 7/7] nvme: add support for enhanced metadata Keith Busch @ 2022-01-26 14:38 ` Klaus Jensen 2022-01-26 16:52 ` Keith Busch 7 siblings, 1 reply; 15+ messages in thread From: Klaus Jensen @ 2022-01-26 14:38 UTC (permalink / raw) To: Keith Busch Cc: linux-nvme, linux-kernel, linux-block, axboe, hch, martin.petersen, colyli, arnd [-- Attachment #1: Type: text/plain, Size: 1545 bytes --] On Jan 24 08:01, Keith Busch wrote: > The NVM Express protocol added enhancements to the data integrity field > formats beyond the T10 defined protection information. A detailed > description of the new formats can be found in the NVMe's NVM Command > Set Specification, section 5.2, available at: > > https://nvmexpress.org/wp-content/uploads/NVM-Command-Set-Specification-1.0b-2021.12.18-Ratified.pdf > > This series implements one possible new format: the CRC64 guard with > 48-bit reference tags. This does not add support for the variable > "storage tag" field. > > The NVMe CRC64 parameters (from Rocksoft) were not implemented in the > kernel, so a software implementation is included in this series based on > the generated table. This series does not include any possible hardware > excelleration (ex: x86's pclmulqdq), so it's not very high performant > right now. > Hi Keith, Tested this on QEMU and (assuming we didnt implement the same bugs) it looks good functionally for separate metadata. However, it should also be able to support PRACT (i.e. pi strip/insert device-side) if nvme_ns_has_pi() is updated to also match on the 16 byte pi tuple. I made it work by just hitting it with a hammer and changing the comparison to hard-coded 16 bytes, but it should of course handle both cases. Naveen and I will post the emulated implementation (that certainly isnt very high performant either) on qemu-block ASAP if others are interested in giving this a spin without having hardware available. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFC 0/7] 64-bit data integrity field support 2022-01-26 14:38 ` [RFC 0/7] 64-bit data integrity field support Klaus Jensen @ 2022-01-26 16:52 ` Keith Busch 0 siblings, 0 replies; 15+ messages in thread From: Keith Busch @ 2022-01-26 16:52 UTC (permalink / raw) To: Klaus Jensen Cc: linux-nvme, linux-kernel, linux-block, axboe, hch, martin.petersen, colyli, arnd On Wed, Jan 26, 2022 at 03:38:13PM +0100, Klaus Jensen wrote: > On Jan 24 08:01, Keith Busch wrote: > > The NVM Express protocol added enhancements to the data integrity field > > formats beyond the T10 defined protection information. A detailed > > description of the new formats can be found in the NVMe's NVM Command > > Set Specification, section 5.2, available at: > > > > https://nvmexpress.org/wp-content/uploads/NVM-Command-Set-Specification-1.0b-2021.12.18-Ratified.pdf > > > > This series implements one possible new format: the CRC64 guard with > > 48-bit reference tags. This does not add support for the variable > > "storage tag" field. > > > > The NVMe CRC64 parameters (from Rocksoft) were not implemented in the > > kernel, so a software implementation is included in this series based on > > the generated table. This series does not include any possible hardware > > excelleration (ex: x86's pclmulqdq), so it's not very high performant > > right now. > > > > Hi Keith, > > Tested this on QEMU and (assuming we didnt implement the same bugs) it > looks good functionally for separate metadata. However, it should also > be able to support PRACT (i.e. pi strip/insert device-side) if > nvme_ns_has_pi() is updated to also match on the 16 byte pi tuple. I > made it work by just hitting it with a hammer and changing the > comparison to hard-coded 16 bytes, but it should of course handle both > cases. Thanks for checking with the qemu device! I'll add the PRACT support for the next version, but will wait till next week to post it in case there's more feedback to consider. > Naveen and I will post the emulated implementation (that certainly isnt > very high performant either) on qemu-block ASAP if others are interested > in giving this a spin without having hardware available. There are more features with this TP than are implemented here. I'm just enabling a product that only supports the 64-bit CRC with 0 STS, but I'm assuming your QEMU implementation may be more feature complete. If anyone is interested in 32-bit CRC or >0 STS, there's more work to follow on from here, but I currently don't have such a target to test against. ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-01-26 16:52 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-01-24 16:01 [RFC 0/7] 64-bit data integrity field support Keith Busch 2022-01-24 16:01 ` [RFC 1/7] block: support pi with extended metadata Keith Busch 2022-01-24 16:01 ` [RFC 2/7] nvme: allow integrity on extended metadata formats Keith Busch 2022-01-24 16:01 ` [RFC 3/7] lib: add rocksoft model crc64 Keith Busch 2022-01-25 5:10 ` Eric Biggers 2022-01-25 15:39 ` Keith Busch 2022-01-24 16:01 ` [RFC 4/7] lib: add crc64 tests Keith Busch 2022-01-25 14:12 ` kernel test robot 2022-01-24 16:01 ` [RFC 5/7] asm-generic: introduce be48 unaligned accessors Keith Busch 2022-01-24 17:38 ` Arnd Bergmann 2022-01-24 16:01 ` [RFC 6/7] block: add pi for nvme enhanced integrity Keith Busch 2022-01-24 16:01 ` [RFC 7/7] nvme: add support for enhanced metadata Keith Busch 2022-01-24 22:23 ` kernel test robot 2022-01-26 14:38 ` [RFC 0/7] 64-bit data integrity field support Klaus Jensen 2022-01-26 16:52 ` Keith Busch
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.