Re: seastar and 'tame reactor'

[Josh Durgin <jdurgin@redhat.com>]

[adding ceph-devel]

On 01/30/2018 01:56 PM, Casey Bodley wrote:
> Hey Josh,
> 
> I heard you mention in the call yesterday that you're looking into this 
> part of seastar integration. I was just reading through the relevant 
> code over the weekend, and wanted to compare notes:
> 
> 
> in seastar, all cross-core communication goes through lockfree spsc 
> queues, which are encapsulated by 'class smp_message_queue' in 
> core/reactor.hh. all of these queues (smp::_qs) are allocated on startup 
> in smp::configure(). early in reactor::run() (which is effectively each 
> seastar thread's entrypoint), it registers a smp_poller to poll all of 
> the queues directed at that cpu
> 
> what we need is a way to inject messages into each seastar reactor from 
> arbitrary/external threads. our requirements are very similar to 
> smp_message_queue's, with a few exceptions:
> 
> -each seastar reactor should be the single consumer of a multi-producer 
> queue, and poll on that as well

Yes, this is what I was thinking too, maybe a boost::lockfree::queue

> -the submit() function would return void instead of a future (which 
> removes the need for a lot of other stuff, like the _completions queue, 
> async_work_item::_promise, etc)
> 
> figuring out how to factor this stuff out of smp_message_queue cleanly 
> is the hard part i guess

I was thinking it could start off as a separate implementation, but
hadn't looked too closely at sharing pieces of it.

> in terms of startup, it could be allocated as a static array similar to 
> smp::_qs (except it would be dimensioned by [smp::count] instead of 
> [smp::count][smp::count]). then a new function could be added alongside 
> smp::submit_to() that submits to the given cpu's external queue (and 
> also returns void). this stuff should probably be disabled by default, 
> and only turned on if enabled in configuration

++

> for a super simple unit test, you could spawn an external thread that 
> does something like this:
> 
> std::mutex mutex;
> std::condition_variable cond;
> std::atomic<int> completions = 0;
> // submit a message to each reactor
> for (int i = 0; i < smp::count; i++) {
>    smp::external_submit_to(i, [&] { ++completions; cond.notify_one(); });
> }
> // wait for all completions
> std::unique_lock lock(mutex);
> cond.wait(lock, [&] { return completions == smp::count; });

Yeah, this looks like a nice example.

> Sorry that I've been slow to help with this - keep me posted?

No worries, I've been slow about this too - I've asked Kefu to look at
it this morning, so I'm sure he'll have some more thoughts soon.

Josh