Re: [SPDK] Blobstore threading

Re: [SPDK] Blobstore threading

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 4227 bytes --]

Thanks, guys. I saw the notes in documentation but I forgot to put the
spdk_bs_init() function on my mgmt thread.

On Tue, Apr 9, 2019 at 2:28 PM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Tue, 2019-04-09 at 20:13 +0000, Zawadzki, Tomasz wrote:
> > Hi Michael,
> >
> > Blobstore does support IO submissions from multiple threads. To
> accomplish
> > that it uses SPDK IO channel abstraction, which in a way can be thought
> of as
> > being associated 1:1 with a thread.
> > https://spdk.io/doc/blob.html
> >
> > Examples of blobstore API usage in such way can be found in logical
> volumes
> > and blobfs.
> >
> > If you suspect that there might be an issue in blobstore, feel free to
> file
> > Github Issue with the crash.
> >
> > Tomek
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> Haeuptle
> > Sent: Tuesday, April 9, 2019 12:14 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Blobstore threading
> >
> > Hello,
> >
> > I've been experimenting with the blobstore and I'm not 100% sure I
> understand
> > the threading model.
> >
> > Does the same blobstore support IO submissions from multiple threads?
> >
> > For example, I've created a simple app that registers a poller on 4
> reactors
> > that each creates a blob, writes to it in 4KB chunks and once the
> cluster size
> > has been reached the blob is closed and deleted. After the delete, the
> > sequence repeats. This is happening in each of the pollers that runs on a
> > reactor.
> > I'm using the same blobstore for the above sequence and a Malloc backend
> > device.
> >
> > The app sometimes cores immediately or after a couple of minutes. There
> seems
> > to be a race somewhere (not ruling out code I added).
> > So I was wondering if the the above is a supported scenario or if the
> > blobstore only works on a single core.
>
> I think there is a lot of room for clarification here. Some operations,
> such as
> I/O to a blob, are safe to do in parallel as Tomek said. Other operations
> are
> not safe. Here is the documentation from the top of blob.h:
>
>  * The blobstore is designed to be very high performance, and thus has
>  * a few general rules regarding thread safety to avoid taking locks
>  * in the I/O path.  This is primarily done by only allowing most
>  * functions to be called on the metadata thread.  The metadata thread is
>  * the thread which called spdk_bs_init() or spdk_bs_load().
>  *
>  * Functions starting with the prefix "spdk_blob_io" are passed a channel
>  * as an argument, and channels may only be used from the thread they were
>  * created on. See \ref spdk_bs_alloc_io_channel.  These are the only
>  * functions that may be called from a thread other than the metadata
>  * thread.
>
> So you need a "management" thread that does the create/open/delete of
> blobs.
> Then, you can allocate as many channels as you'd like (one per thread) and
> perform I/O operations in parallel. I believe if you run a debug build of
> SPDK,
> there are asserts that will verify you're following the rules.
>
> Thanks,
> Ben
>
> >
> > Ultimately, I want to create and read blobs via a socket interface and
> support
> > a large number of parallel requests from multiple clients. For blob
> reads, I
> > was thinking to simply let the blobstore read call back send the chunk
> back
> > over the socket to the client.
> > If the blobstore can only run on one core then I probably would have to
> submit
> > events to the blobstore thread from the socket threads or something along
> > those lines which could impact scalability.
> >
> > Thanks.
> >
> > -- Michael
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] Blobstore threading

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 3670 bytes --]

On Tue, 2019-04-09 at 20:13 +0000, Zawadzki, Tomasz wrote:
> Hi Michael,
> 
> Blobstore does support IO submissions from multiple threads. To accomplish
> that it uses SPDK IO channel abstraction, which in a way can be thought of as
> being associated 1:1 with a thread.
> https://spdk.io/doc/blob.html
> 
> Examples of blobstore API usage in such way can be found in logical volumes
> and blobfs.
> 
> If you suspect that there might be an issue in blobstore, feel free to file
> Github Issue with the crash.
> 
> Tomek
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael Haeuptle
> Sent: Tuesday, April 9, 2019 12:14 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Blobstore threading
> 
> Hello,
> 
> I've been experimenting with the blobstore and I'm not 100% sure I understand
> the threading model.
> 
> Does the same blobstore support IO submissions from multiple threads?
> 
> For example, I've created a simple app that registers a poller on 4 reactors
> that each creates a blob, writes to it in 4KB chunks and once the cluster size
> has been reached the blob is closed and deleted. After the delete, the
> sequence repeats. This is happening in each of the pollers that runs on a
> reactor.
> I'm using the same blobstore for the above sequence and a Malloc backend
> device.
> 
> The app sometimes cores immediately or after a couple of minutes. There seems
> to be a race somewhere (not ruling out code I added).
> So I was wondering if the the above is a supported scenario or if the
> blobstore only works on a single core.

I think there is a lot of room for clarification here. Some operations, such as
I/O to a blob, are safe to do in parallel as Tomek said. Other operations are
not safe. Here is the documentation from the top of blob.h:

 * The blobstore is designed to be very high performance, and thus has
 * a few general rules regarding thread safety to avoid taking locks
 * in the I/O path.  This is primarily done by only allowing most
 * functions to be called on the metadata thread.  The metadata thread is
 * the thread which called spdk_bs_init() or spdk_bs_load().
 *
 * Functions starting with the prefix "spdk_blob_io" are passed a channel
 * as an argument, and channels may only be used from the thread they were
 * created on. See \ref spdk_bs_alloc_io_channel.  These are the only
 * functions that may be called from a thread other than the metadata
 * thread.

So you need a "management" thread that does the create/open/delete of blobs.
Then, you can allocate as many channels as you'd like (one per thread) and
perform I/O operations in parallel. I believe if you run a debug build of SPDK,
there are asserts that will verify you're following the rules.

Thanks,
Ben

> 
> Ultimately, I want to create and read blobs via a socket interface and support
> a large number of parallel requests from multiple clients. For blob reads, I
> was thinking to simply let the blobstore read call back send the chunk back
> over the socket to the client.
> If the blobstore can only run on one core then I probably would have to submit
> events to the blobstore thread from the socket threads or something along
> those lines which could impact scalability.
> 
> Thanks.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Blobstore threading

[Zawadzki, Tomasz]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]

Hi Michael,

Blobstore does support IO submissions from multiple threads. To accomplish that it uses SPDK IO channel abstraction, which in a way can be thought of as being associated 1:1 with a thread.
https://spdk.io/doc/blob.html

Examples of blobstore API usage in such way can be found in logical volumes and blobfs.

If you suspect that there might be an issue in blobstore, feel free to file Github Issue with the crash.

Tomek

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael Haeuptle
Sent: Tuesday, April 9, 2019 12:14 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Blobstore threading

Hello,

I've been experimenting with the blobstore and I'm not 100% sure I understand the threading model.

Does the same blobstore support IO submissions from multiple threads?

For example, I've created a simple app that registers a poller on 4 reactors that each creates a blob, writes to it in 4KB chunks and once the cluster size has been reached the blob is closed and deleted. After the delete, the sequence repeats. This is happening in each of the pollers that runs on a reactor.
I'm using the same blobstore for the above sequence and a Malloc backend device.

The app sometimes cores immediately or after a couple of minutes. There seems to be a race somewhere (not ruling out code I added).
So I was wondering if the the above is a supported scenario or if the blobstore only works on a single core.

Ultimately, I want to create and read blobs via a socket interface and support a large number of parallel requests from multiple clients. For blob reads, I was thinking to simply let the blobstore read call back send the chunk back over the socket to the client.
If the blobstore can only run on one core then I probably would have to submit events to the blobstore thread from the socket threads or something along those lines which could impact scalability.

Thanks.

-- Michael
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

[SPDK] Blobstore threading

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

Hello,

I've been experimenting with the blobstore and I'm not 100% sure I
understand the threading model.

Does the same blobstore support IO submissions from multiple threads?

For example, I've created a simple app that registers a poller on 4
reactors that each creates a blob, writes to it in 4KB chunks and once the
cluster size has been reached the blob is closed and deleted. After the
delete, the sequence repeats. This is happening in each of the pollers that
runs on a reactor.
I'm using the same blobstore for the above sequence and a Malloc backend
device.

The app sometimes cores immediately or after a couple of minutes. There
seems to be a race somewhere (not ruling out code I added).
So I was wondering if the the above is a supported scenario or if the
blobstore only works on a single core.

Ultimately, I want to create and read blobs via a socket interface and
support a large number of parallel requests from multiple clients. For blob
reads, I was thinking to simply let the blobstore read call back send the
chunk back over the socket to the client.
If the blobstore can only run on one core then I probably would have to
submit events to the blobstore thread from the socket threads or something
along those lines which could impact scalability.

Thanks.

-- Michael

Re: [SPDK] BlobStore Threading

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]



From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Fenggang Wu <wuxx0835(a)umn.edu>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, March 8, 2018 at 12:06 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] BlobStore Threading

Hi,

I am studying the blobstore threading model. Considering a scenario that a RocksDB is sitting above blobstore. There are 4 reading threads and 1 writing thread.

Correct me if I get it wrong: there is a dedicated Async thread spawned by SPDK, receiving message from RocksDB threads that calls the spdk_file_{read|write} API. Such rocksdb threads performing I/O will consist of the 4 reading thread, some WAL writing thread, memtable flushing thread and compaction thread. The spdk_file_{read|write} are sync APIs, in which the caller thread will wait on semaphore after sending the I/O request to the dedicated I/O thread, until waken up from the semaphore.

Here are my questions:

1) Which thread is signaling the semaphore and wakes up the sleeping caller thread? I guess it is not the dedicated I/O thread, in that it is busying receiving request from the message queue and polling all the I/O devices.

2) Or more generally, which thread is executing the call back functions embedded in the I/O request message? Does the dedicated I/O thread spawn a SPDK thread right away just to execute the call back? Is there a thread pool to allocate from? Or, is there any limit on the numbers of the spawned SPDK threads?

Thank you very much!
-Fenggang


Hi Fenggang,

It is the SPDK async thread that signals the semaphore to wake the sleeping caller thread.  It only signals the caller thread after any associated asynchronous I/O operations are complete.

-Jim




[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5590 bytes --]

[SPDK] BlobStore Threading

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 1282 bytes --]

Hi,

I am studying the blobstore threading model. Considering a scenario that a
RocksDB is sitting above blobstore. There are 4 reading threads and 1
writing thread.

Correct me if I get it wrong: there is a dedicated Async thread spawned by
SPDK, receiving message from RocksDB threads that calls the
spdk_file_{read|write} API. Such rocksdb threads performing I/O will
consist of the 4 reading thread, some WAL writing thread, memtable flushing
thread and compaction thread. The spdk_file_{read|write} are sync APIs, in
which the caller thread will wait on semaphore after sending the I/O
request to the dedicated I/O thread, until waken up from the semaphore.

Here are my questions:

1) Which thread is signaling the semaphore and wakes up the sleeping caller
thread? I guess it is not the dedicated I/O thread, in that it is busying
receiving request from the message queue and polling all the I/O devices.

2) Or more generally, which thread is executing the call back functions
embedded in the I/O request message? Does the dedicated I/O thread spawn a
SPDK thread right away just to execute the call back? Is there a thread
pool to allocate from? Or, is there any limit on the numbers of the spawned
SPDK threads?

Thank you very much!
-Fenggang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1461 bytes --]