* parallel transaction submit
@ 2016-08-25 7:45 Tang, Haodong
2016-08-25 7:55 ` Haomai Wang
0 siblings, 1 reply; 6+ messages in thread
From: Tang, Haodong @ 2016-08-25 7:45 UTC (permalink / raw)
To: sweil, varada.kari, ceph-devel
Hi Sage, Varada
Noticed you are making parallel transaction submits, we also worked out a prototype that looks similar, here is the link for the implementation: https://github.com/ceph/ceph/pull/10856
Background:
From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
Implementation:
The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
Performance evaluation:
Test ENV:
4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
Performance:
We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
Test summary:
QD Scaling Test - 4k Random Write:
QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
With rocksdb (IOPS) 682 173000 190000 203000 204000
With memdb (IOPS) 704 180000 194000 206000 218000
With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
It seems single thread of transaction submits will be a bottleneck if using MemDB.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: parallel transaction submit
2016-08-25 7:45 parallel transaction submit Tang, Haodong
@ 2016-08-25 7:55 ` Haomai Wang
2016-08-25 8:47 ` Varada Kari
2016-08-25 8:48 ` Tang, Haodong
0 siblings, 2 replies; 6+ messages in thread
From: Haomai Wang @ 2016-08-25 7:55 UTC (permalink / raw)
To: Tang, Haodong; +Cc: sweil, varada.kari, ceph-devel
looks very litlle improvements. rocksdb result meet my expectation
because rocksdb internal has lock for multi sync write. But memdb
improments is a little confusing.
On Thu, Aug 25, 2016 at 3:45 PM, Tang, Haodong <haodong.tang@intel.com> wrote:
> Hi Sage, Varada
>
> Noticed you are making parallel transaction submits, we also worked out a prototype that looks similar, here is the link for the implementation: https://github.com/ceph/ceph/pull/10856
>
> Background:
> From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
>
> Implementation:
> The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
>
> Performance evaluation:
> Test ENV:
> 4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
> Performance:
> We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
>
> What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
>
> Test summary:
> QD Scaling Test - 4k Random Write:
> QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
> With rocksdb (IOPS) 682 173000 190000 203000 204000
> With memdb (IOPS) 704 180000 194000 206000 218000
> With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
> With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
>
>
> It seems single thread of transaction submits will be a bottleneck if using MemDB.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: parallel transaction submit
2016-08-25 7:55 ` Haomai Wang
@ 2016-08-25 8:47 ` Varada Kari
2016-08-25 14:11 ` Sage Weil
2016-08-25 8:48 ` Tang, Haodong
1 sibling, 1 reply; 6+ messages in thread
From: Varada Kari @ 2016-08-25 8:47 UTC (permalink / raw)
To: Haomai Wang, Tang, Haodong; +Cc: sweil, ceph-devel
Hi,
Increasing the number of the kv_sync_threads is not giving much of
performance. In the current threading model, shard_worker submits the IO
to the block device which are handled by aio_callback thread(which is
one) and submits to the kv_sync thread, which batches the requests and
submits to the rocksdb. Because kv_sync batches the requests and submits
the requests, we might observe more time spent on kv_sync_thread
routine. And i haven't observed much of an improvement by adding more
threads here.
But when increased the number of callbacks thread from aio(still needs
some refinements in polling for the request completions) and completing
the write completion in the same thread context increased some
performance. I don't have the numbers to say how much, but that is
better than having multiple kv_sync threads, adding one more queue and
lock. You can refer to
https://github.com/varadakari/ceph/commits/wip-parallel-aiocb (ignore
the first commit, was trying to do sync transaction in the same thread
context of sharded worker to measure the latency).
was exploring a way to have the aio callback thread matached/reserved at
the time of io submission, so that we don't need to do io_getevents(),
kind of a async callback to the specified thread so that we can avoid
some waiting logic in io_getevents() and process the request in the same
thread context. You can refer to
http://manpages.ubuntu.com/manpages/wily/man3/io_set_callback.3.html. I
don't have the working code ready for this. FWIW, that is worth
experimenting and see if it reduces any latency.
Varada
On Thursday 25 August 2016 01:25 PM, Haomai Wang wrote:
> looks very litlle improvements. rocksdb result meet my expectation
> because rocksdb internal has lock for multi sync write. But memdb
> improments is a little confusing.
>
> On Thu, Aug 25, 2016 at 3:45 PM, Tang, Haodong <haodong.tang@intel.com> wrote:
>> Hi Sage, Varada
>>
>> Noticed you are making parallel transaction submits, we also worked out a prototype that looks similar, here is the link for the implementation: https://github.com/ceph/ceph/pull/10856
>>
>> Background:
>> From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
>>
>> Implementation:
>> The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
>>
>> Performance evaluation:
>> Test ENV:
>> 4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
>> Performance:
>> We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
>>
>> What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
>>
>> Test summary:
>> QD Scaling Test - 4k Random Write:
>> QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
>> With rocksdb (IOPS) 682 173000 190000 203000 204000
>> With memdb (IOPS) 704 180000 194000 206000 218000
>> With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
>> With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
>>
>>
>> It seems single thread of transaction submits will be a bottleneck if using MemDB.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: parallel transaction submit
2016-08-25 7:55 ` Haomai Wang
2016-08-25 8:47 ` Varada Kari
@ 2016-08-25 8:48 ` Tang, Haodong
1 sibling, 0 replies; 6+ messages in thread
From: Tang, Haodong @ 2016-08-25 8:48 UTC (permalink / raw)
To: Haomai Wang; +Cc: sweil, varada.kari, ceph-devel
Yeah, from the perf counter, we do reduce time waiting for kv_queue, but increase rocksdb submisstion time. Seems it is better to do batch submission for rocksdb instead of parallel submission. Besides, parallel submission also make bdev->flush more frequent.
From the test result, there is a little performance improvement if using memdb, but compared with rocksdb, throughput is more stable.
-----Original Message-----
From: Haomai Wang [mailto:haomai@xsky.com]
Sent: Thursday, August 25, 2016 3:55 PM
To: Tang, Haodong <haodong.tang@intel.com>
Cc: sweil@redhat.com; varada.kari@sandisk.com; ceph-devel@vger.kernel.org
Subject: Re: parallel transaction submit
looks very litlle improvements. rocksdb result meet my expectation because rocksdb internal has lock for multi sync write. But memdb improments is a little confusing.
On Thu, Aug 25, 2016 at 3:45 PM, Tang, Haodong <haodong.tang@intel.com> wrote:
> Hi Sage, Varada
>
> Noticed you are making parallel transaction submits, we also worked
> out a prototype that looks similar, here is the link for the
> implementation: https://github.com/ceph/ceph/pull/10856
>
> Background:
> From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
>
> Implementation:
> The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
>
> Performance evaluation:
> Test ENV:
> 4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
> Performance:
> We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
>
> What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
>
> Test summary:
> QD Scaling Test - 4k Random Write:
> QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
> With rocksdb (IOPS) 682 173000 190000 203000 204000
> With memdb (IOPS) 704 180000 194000 206000 218000
> With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
> With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
>
>
> It seems single thread of transaction submits will be a bottleneck if using MemDB.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: parallel transaction submit
2016-08-25 8:47 ` Varada Kari
@ 2016-08-25 14:11 ` Sage Weil
2016-08-25 14:26 ` Varada Kari
0 siblings, 1 reply; 6+ messages in thread
From: Sage Weil @ 2016-08-25 14:11 UTC (permalink / raw)
To: Varada Kari; +Cc: Haomai Wang, Tang, Haodong, ceph-devel
On Thu, 25 Aug 2016, Varada Kari wrote:
> Hi,
>
> Increasing the number of the kv_sync_threads is not giving much of
> performance. In the current threading model, shard_worker submits the IO
> to the block device which are handled by aio_callback thread(which is
> one) and submits to the kv_sync thread, which batches the requests and
> submits to the rocksdb. Because kv_sync batches the requests and submits
> the requests, we might observe more time spent on kv_sync_thread
> routine. And i haven't observed much of an improvement by adding more
> threads here.
>
> But when increased the number of callbacks thread from aio(still needs
> some refinements in polling for the request completions) and completing
> the write completion in the same thread context increased some
> performance. I don't have the numbers to say how much, but that is
> better than having multiple kv_sync threads, adding one more queue and
> lock. You can refer to
> https://github.com/varadakari/ceph/commits/wip-parallel-aiocb (ignore
> the first commit, was trying to do sync transaction in the same thread
> context of sharded worker to measure the latency).
Yeah, I think this is right. I see two avenues of attack:
- Try to eliminate the handoff to _kv_sync_thread by having the
transaction submitted to rocksdb in the calling thread. This will
require a bit of refactoring but I think it's possible. We don't actually
want to block, though, so it'll be an async submission, and we'll still
need kv_sync_thread just telling rocksdb to commit in a loop and
triggering callbacks. A recent PR sharded the completion finishers so I'm
guessing the final step would be some affinity thing that pins the
finishers to the same cores as the submitters?
- Shard the io completion (before we submit the kv transaction). Not sure
if we want a thread per shard, or polls at opportunistic/strategic points
in code. The goal would be keeping the processing local to the
core/socket (vs the current strategy of a single thread waiting/polling
for completions and doing the next phase of work).
> was exploring a way to have the aio callback thread matached/reserved at
> the time of io submission, so that we don't need to do io_getevents(),
> kind of a async callback to the specified thread so that we can avoid
> some waiting logic in io_getevents() and process the request in the same
> thread context. You can refer to
> http://manpages.ubuntu.com/manpages/wily/man3/io_set_callback.3.html. I
> don't have the working code ready for this. FWIW, that is worth
> experimenting and see if it reduces any latency.
I don't think this will help--it just means you're using a layer of the
library that's calling getevents for you and calling your callback.
Thanks!
sage
>
> Varada
>
> On Thursday 25 August 2016 01:25 PM, Haomai Wang wrote:
> > looks very litlle improvements. rocksdb result meet my expectation
> > because rocksdb internal has lock for multi sync write. But memdb
> > improments is a little confusing.
> >
> > On Thu, Aug 25, 2016 at 3:45 PM, Tang, Haodong <haodong.tang@intel.com> wrote:
> >> Hi Sage, Varada
> >>
> >> Noticed you are making parallel transaction submits, we also worked out a prototype that looks similar, here is the link for the implementation: https://github.com/ceph/ceph/pull/10856
> >>
> >> Background:
> >> From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
> >>
> >> Implementation:
> >> The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
> >>
> >> Performance evaluation:
> >> Test ENV:
> >> 4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
> >> Performance:
> >> We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
> >>
> >> What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
> >>
> >> Test summary:
> >> QD Scaling Test - 4k Random Write:
> >> QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
> >> With rocksdb (IOPS) 682 173000 190000 203000 204000
> >> With memdb (IOPS) 704 180000 194000 206000 218000
> >> With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
> >> With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
> >>
> >>
> >> It seems single thread of transaction submits will be a bottleneck if using MemDB.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: parallel transaction submit
2016-08-25 14:11 ` Sage Weil
@ 2016-08-25 14:26 ` Varada Kari
0 siblings, 0 replies; 6+ messages in thread
From: Varada Kari @ 2016-08-25 14:26 UTC (permalink / raw)
To: Sage Weil; +Cc: Haomai Wang, Tang, Haodong, ceph-devel
On Thursday 25 August 2016 07:41 PM, Sage Weil wrote:
> On Thu, 25 Aug 2016, Varada Kari wrote:
>> Hi,
>>
>> Increasing the number of the kv_sync_threads is not giving much of
>> performance. In the current threading model, shard_worker submits the IO
>> to the block device which are handled by aio_callback thread(which is
>> one) and submits to the kv_sync thread, which batches the requests and
>> submits to the rocksdb. Because kv_sync batches the requests and submits
>> the requests, we might observe more time spent on kv_sync_thread
>> routine. And i haven't observed much of an improvement by adding more
>> threads here.
>>
>> But when increased the number of callbacks thread from aio(still needs
>> some refinements in polling for the request completions) and completing
>> the write completion in the same thread context increased some
>> performance. I don't have the numbers to say how much, but that is
>> better than having multiple kv_sync threads, adding one more queue and
>> lock. You can refer to
>> https://github.com/varadakari/ceph/commits/wip-parallel-aiocb (ignore
>> the first commit, was trying to do sync transaction in the same thread
>> context of sharded worker to measure the latency).
> Yeah, I think this is right. I see two avenues of attack:
>
> - Try to eliminate the handoff to _kv_sync_thread by having the
> transaction submitted to rocksdb in the calling thread. This will
> require a bit of refactoring but I think it's possible. We don't actually
> want to block, though, so it'll be an async submission, and we'll still
> need kv_sync_thread just telling rocksdb to commit in a loop and
> triggering callbacks. A recent PR sharded the completion finishers so I'm
> guessing the final step would be some affinity thing that pins the
> finishers to the same cores as the submitters?
I kind of copied what kv_sync_thread does and was able to run multiple
callbacks at the same time.
If we can complete the write ack in the same aio_cb thread not handing
off to finisher thread, we can eliminate
one thread switch and we can have same number of threads as shards. We
can use the same logic to handle the finishers
(not sure if we can process the request by reading the osr here) for the
callback threads.
Varada
> - Shard the io completion (before we submit the kv transaction). Not sure
> if we want a thread per shard, or polls at opportunistic/strategic points
> in code. The goal would be keeping the processing local to the
> core/socket (vs the current strategy of a single thread waiting/polling
> for completions and doing the next phase of work).
>
>> was exploring a way to have the aio callback thread matached/reserved at
>> the time of io submission, so that we don't need to do io_getevents(),
>> kind of a async callback to the specified thread so that we can avoid
>> some waiting logic in io_getevents() and process the request in the same
>> thread context. You can refer to
>> http://manpages.ubuntu.com/manpages/wily/man3/io_set_callback.3.html. I
>> don't have the working code ready for this. FWIW, that is worth
>> experimenting and see if it reduces any latency.
> I don't think this will help--it just means you're using a layer of the
> library that's calling getevents for you and calling your callback.
>
> Thanks!
> sage
>
>
>> Varada
>>
>> On Thursday 25 August 2016 01:25 PM, Haomai Wang wrote:
>>> looks very litlle improvements. rocksdb result meet my expectation
>>> because rocksdb internal has lock for multi sync write. But memdb
>>> improments is a little confusing.
>>>
>>> On Thu, Aug 25, 2016 at 3:45 PM, Tang, Haodong <haodong.tang@intel.com> wrote:
>>>> Hi Sage, Varada
>>>>
>>>> Noticed you are making parallel transaction submits, we also worked out a prototype that looks similar, here is the link for the implementation: https://github.com/ceph/ceph/pull/10856
>>>>
>>>> Background:
>>>> From the perf counter we added, found it spent a lot time in kv_queue, that is, single thread transaction submits is not competent to handle the transaction from OSD.
>>>>
>>>> Implementation:
>>>> The key thought is to use multiple thread and assign each TransContext to one of the processing threads. In order to parallelize transaction submit, add different kv_locks and kv_conds for each thread.
>>>>
>>>> Performance evaluation:
>>>> Test ENV:
>>>> 4 x server, 4 x client, 16 x Intel S3700 as block device, and 4 x Intel P3600 as Rocksdb/WAL device.
>>>> Performance:
>>>> We also did several quick tests to verify the performance benefit, the results showed that parallel transaction submission will brought 10% performance improvement if using memdb, but little performance improvement with rocksdb.
>>>>
>>>> What's more, without parallel transaction submits, we also see performance boost if just changing to MemDB, but a little.
>>>>
>>>> Test summary:
>>>> QD Scaling Test - 4k Random Write:
>>>> QD = 1 QD = 16 QD = 32 QD = 64 QD = 128
>>>> With rocksdb (IOPS) 682 173000 190000 203000 204000
>>>> With memdb (IOPS) 704 180000 194000 206000 218000
>>>> With rocksdb+multiple_kv_thread(IOPS) / 164243 167037 180961 201752
>>>> With memdb+multiple_kv_thread(IOPS) / 176000 200000 221000 227000
>>>>
>>>>
>>>> It seems single thread of transaction submits will be a bottleneck if using MemDB.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-08-25 14:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-25 7:45 parallel transaction submit Tang, Haodong
2016-08-25 7:55 ` Haomai Wang
2016-08-25 8:47 ` Varada Kari
2016-08-25 14:11 ` Sage Weil
2016-08-25 14:26 ` Varada Kari
2016-08-25 8:48 ` Tang, Haodong
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.