* [SPDK] Bug in BlobFS
@ 2023-05-06 8:09 韩光阳
2023-05-11 22:48 ` Harris, James R
0 siblings, 1 reply; 2+ messages in thread
From: 韩光阳 @ 2023-05-06 8:09 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 879 bytes --]
Hello, we seems to find a fatal bug in BlobFS, and need your confirm. Thanks!
In my last email, I said we encountered with a weired bug. Rocksdb reported "Corruption: block checksum mismatch" when using spdk blobfs to manage IO.
After monthly debug, we found BlobFS should be blame. In FUNC __read_done, it regard less of the incoming arg "bserrno", calling __rw_done(req, 0) with errno 0 directly.
Actually, we found FUNC spdk_blob_io_read may fail due to lack of resources, including running out of req in Blobfs sync dev or req in bs_channel. Blobfs should reported ENOMEM to Rocksdb, but calling __rw_done(req, 0) means no error, so Rocksdb calculate CSUM on a nonsense buf, leading to "Corruption: block checksum mismatch".
By searching modify history, we found it is the original code when blobfs was introduced, is that really a bug, uh-huh?
[-- Attachment #2: Type: text/html, Size: 1049 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [SPDK] Bug in BlobFS
2023-05-06 8:09 [SPDK] Bug in BlobFS 韩光阳
@ 2023-05-11 22:48 ` Harris, James R
0 siblings, 0 replies; 2+ messages in thread
From: Harris, James R @ 2023-05-11 22:48 UTC (permalink / raw)
To: 韩光阳, spdk
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb2312", Size: 1925 bytes --]
Hi Guangyang,
Yes, this does look like a bug.
__read_done() should first do a check like this:
if (bserrno != 0) {
__rw_done(req, bserrno);
return;
}
This will at least report the error correctly up the stack.
But it also looks like at least for your workload, default number of max_channel_ops is not sufficient for the underlying blobstore. Can you describe your workload in more detail? By default, it supports up to 512 I/O at a time ¨C this can be increased but I¡¯d like to understand the configuration and workload that is generating more I/O than this.
Note, you can increase this number by adding the following to spdk_fs_load():
spdk_bs_opts_init(&bs_opts, sizeof(bs_opts));
>>> bs_opts.max_channel_ops *= 4; /* Allocate 4x as many ops as usual. */
bs_opts.iter_cb_fn = iter_cb;
Regards,
Jim
From: SPDK <spdk-bounces@lists.linuxfoundation.org> on behalf of º«¹âÑô <guangyang.han@foxmail.com>
Date: Thursday, May 11, 2023 at 3:34 PM
To: spdk <spdk@lists.linuxfoundation.org>
Subject: [SPDK] Bug in BlobFS
Hello, we seems to find a fatal bug in BlobFS, and need your confirm. Thanks!
In my last email, I said we encountered with a weired bug. Rocksdb reported "Corruption: block checksum mismatch" when using spdk blobfs to manage IO.
After monthly debug, we found BlobFS should be blame. In FUNC __read_done, it regard less of the incoming arg "bserrno", calling __rw_done(req, 0) with errno 0 directly.
Actually, we found FUNC spdk_blob_io_read may fail due to lack of resources, including running out of req in Blobfs sync dev or req in bs_channel. Blobfs should reported ENOMEM to Rocksdb, but calling __rw_done(req, 0) means no error, so Rocksdb calculate CSUM on a nonsense buf, leading to "Corruption: block checksum mismatch".
By searching modify history, we found it is the original code when blobfs was introduced, is that really a bug, uh-huh?
[-- Attachment #2: Type: text/html, Size: 7158 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-05-11 22:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-06 8:09 [SPDK] Bug in BlobFS 韩光阳
2023-05-11 22:48 ` Harris, James R
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.