io-uring.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
To: Yu Jian Wu <yujian.wu1@gmail.com>,
	io-uring@vger.kernel.org, "axboe@kernel.dk" <axboe@kernel.dk>,
	joseph qi <joseph.qi@linux.alibaba.com>,
	asaf.cidon@columbia.edu, stutsman@cs.utah.edu
Subject: Re: Should io_sq_thread belongs to specific cpu, not io_uring instance
Date: Wed, 15 Apr 2020 20:18:24 +0800	[thread overview]
Message-ID: <c94098d2-279e-a552-91ec-8a8f177d770a@linux.alibaba.com> (raw)
In-Reply-To: <e876a7e8-982b-a193-f4dc-56e7e990b7c5@gmail.com>

hi,

> 
> On 4/14/20 9:08 AM, Xiaoguang Wang wrote:
>> hi,
>>
>> Currently we can create multiple io_uring instances which all have SQPOLL
>> enabled and make them run in the same cpu core by setting sq_thread_cpu
>> argument, but I think this behaviour maybe not efficient. Say we create two
>> io_uring instances, which both have sq_thread_cpu set to 1 and sq_thread_idle
>> set to 1000 milliseconds, there maybe such scene below:
>>    For example, in 0-1s time interval, io_uring instance0 has neither sqes
>> nor cqes, so it just busy waits for new sqes in 0-1s time interval, but
>> io_uring instance1 have work to do, submitting sqes or polling issued requests,
>> then io_uring instance0 will impact io_uring instance1. Of cource io_uring
>> instance1 may impact iouring instance0 as well, which is not efficient. I think
>> the complete disorder of multiple io_uring instances running in same cpu core is
>> not good.
>>
>> How about we create one io_sq_thread for user specified cpu for multiple io_uring
>> instances which try to share this cpu core, that means this io_sq_thread does not
>> belong to specific io_uring instance, it belongs to specific cpu and will
>> handle requests from mulpile io_uring instance, see simple running flow:
>>    1, for cpu 1, now there are no io_uring instances bind to it, so do not create io_sq_thread
>>    2, io_uring instance1 is created and bind to cpu 1, then create cpu1's io_sq_thread
>>    3, io_sq_thread will handle io_uring instance1's requests
>>    4, io_uring instance2 is created and bind to cpu 1, since there are already an
>>       io_sq_thread for cpu 1, will not create an io_sq_thread for cpu1.
>>    5. now io_sq_thread in cpu1 will handle both io_uring instances' requests.
>>
>> What do you think about it? Thanks.
>>
>> Regards,
>> Xiaoguang Wang
>>
> Hi Xiaoguang,
> 
> We (a group of researchers at Utah and Columbia) are currently trying that right now.
Cool, thanks, let me explain more why we need this feature :)
Cpu is a much more important resource. Say a physical machine has 96 cores,
if we run many io_uring instances which all have sqpoll enabled, indeed we
can only allocate a small number of cpus to io_sq_thread, so sharing cpu to
poll is valuable.

> 
> We have an initial prototype going, and we are assessing the performance impact now to see if we can see gains. Basically, have a rcu-list of io_uring_ctx and then traverse the list and do work in a shared io_sq_thread. We are starting experiments on a machine with fast SSDs where we hope to see some performance benefits.
You can try this test case to assessing the performance :)
   1. create two io_uring instances, which both have sqpoll enabled, set
sq_thread_idle to 1000ms and bind to same cpu core.
   2. one io_uring instance just sends one io request per 500ms, which will
make this instance's io_sq_thead always contend for the cpu.
   3. another io_uring instance issues io requests continually, so this
instance's io_sq_thread will also contend for the cpu.
In current io_uring implementation, I think the second io_uring instance will
be impacted by the first io_uring instance.

> 
> We will send the list of patches soon, once we are sure the approach works and we finish cleaning it up. (There is a subtlety of what to do with the timeouts and resched() when not pinning.)
> 
> We'll keep you in the loop on any updates. Feel free to contact any of us.
OK, thanks.

Regards,
Xiaoguang Wang
> 
> Thanks,
> 
> Yu Jian Wu
> 

      reply	other threads:[~2020-04-15 12:18 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-14 13:08 Should io_sq_thread belongs to specific cpu, not io_uring instance Xiaoguang Wang
2020-04-14 18:55 ` Yu Jian Wu
2020-04-15 12:18   ` Xiaoguang Wang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c94098d2-279e-a552-91ec-8a8f177d770a@linux.alibaba.com \
    --to=xiaoguang.wang@linux.alibaba.com \
    --cc=asaf.cidon@columbia.edu \
    --cc=axboe@kernel.dk \
    --cc=io-uring@vger.kernel.org \
    --cc=joseph.qi@linux.alibaba.com \
    --cc=stutsman@cs.utah.edu \
    --cc=yujian.wu1@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).