[SPDK] Bug in BlobFS

[SPDK] Bug in BlobFS

[韩光阳]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Hello, we seems to find a fatal bug in BlobFS, and need your confirm. Thanks!


In my last email, I said we encountered with a weired bug. Rocksdb reported "Corruption: block checksum mismatch" when using spdk blobfs to manage IO. 


After monthly debug, we found BlobFS should be blame. In FUNC __read_done, it regard less of the incoming arg "bserrno", calling __rw_done(req, 0) with errno 0 directly.


Actually, we found FUNC spdk_blob_io_read may fail due to lack of resources, including running out of req in Blobfs sync dev or req in bs_channel. Blobfs should reported ENOMEM to Rocksdb, but calling __rw_done(req, 0) means no error, so Rocksdb calculate CSUM on a nonsense buf, leading to "Corruption: block checksum mismatch".


By searching modify history, we found it is the original code when blobfs was introduced, is that really a bug, uh-huh?

[-- Attachment #2: Type: text/html, Size: 1049 bytes --]

Re: [SPDK] Bug in BlobFS

[Harris, James R]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb2312", Size: 1925 bytes --]

Hi Guangyang,

Yes, this does look like a bug.

__read_done() should first do a check like this:

if (bserrno != 0) {
  __rw_done(req, bserrno);
  return;
}

This will at least report the error correctly up the stack.

But it also looks like at least for your workload, default number of max_channel_ops is not sufficient for the underlying blobstore.  Can you describe your workload in more detail?  By default, it supports up to 512 I/O at a time ¨C this can be increased but I¡¯d like to understand the configuration and workload that is generating more I/O than this.

Note, you can increase this number by adding the following to spdk_fs_load():

spdk_bs_opts_init(&bs_opts, sizeof(bs_opts));
>>> bs_opts.max_channel_ops *= 4; /* Allocate 4x as many ops as usual. */
bs_opts.iter_cb_fn = iter_cb;

Regards,

Jim



From: SPDK <spdk-bounces@lists.linuxfoundation.org> on behalf of º«¹âÑô <guangyang.han@foxmail.com>
Date: Thursday, May 11, 2023 at 3:34 PM
To: spdk <spdk@lists.linuxfoundation.org>
Subject: [SPDK] Bug in BlobFS
Hello, we seems to find a fatal bug in BlobFS, and need your confirm. Thanks!

In my last email, I said we encountered with a weired bug. Rocksdb reported "Corruption: block checksum mismatch" when using spdk blobfs to manage IO.

After monthly debug, we found BlobFS should be blame. In FUNC __read_done, it regard less of the incoming arg "bserrno", calling __rw_done(req, 0) with errno 0 directly.

Actually, we found FUNC spdk_blob_io_read may fail due to lack of resources, including running out of req in Blobfs sync dev or req in bs_channel. Blobfs should reported ENOMEM to Rocksdb, but calling __rw_done(req, 0) means no error, so Rocksdb calculate CSUM on a nonsense buf, leading to "Corruption: block checksum mismatch".

By searching modify history, we found it is the original code when blobfs was introduced, is that really a bug, uh-huh?

[-- Attachment #2: Type: text/html, Size: 7158 bytes --]