From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Fenggang Wu <wuxx0835(a)umn.edu>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, March 8, 2018 at 12:06 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] BlobStore Threading

Hi,

I am studying the blobstore threading model. Considering a scenario that a RocksDB is sitting above blobstore. There are 4 reading threads and 1 writing thread.

Correct me if I get it wrong: there is a dedicated Async thread spawned by SPDK, receiving message from RocksDB threads that calls the spdk_file_{read|write} API. Such rocksdb threads performing I/O will consist of the 4 reading thread, some WAL writing thread, memtable flushing thread and compaction thread. The spdk_file_{read|write} are sync APIs, in which the caller thread will wait on semaphore after sending the I/O request to the dedicated I/O thread, until waken up from the semaphore.

Here are my questions:

1) Which thread is signaling the semaphore and wakes up the sleeping caller thread? I guess it is not the dedicated I/O thread, in that it is busying receiving request from the message queue and polling all the I/O devices.

2) Or more generally, which thread is executing the call back functions embedded in the I/O request message? Does the dedicated I/O thread spawn a SPDK thread right away just to execute the call back? Is there a thread pool to allocate from? Or, is there any limit on the numbers of the spawned SPDK threads?

Thank you very much!
-Fenggang


Hi Fenggang,

It is the SPDK async thread that signals the semaphore to wake the sleeping caller thread.  It only signals the caller thread after any associated asynchronous I/O operations are complete.

-Jim