* FileStore performance: coalescing operations
@ 2015-02-26 14:28 Andreas Bluemle
2015-02-26 15:02 ` Haomai Wang
2015-03-04 1:05 ` Sage Weil
0 siblings, 2 replies; 9+ messages in thread
From: Andreas Bluemle @ 2015-02-26 14:28 UTC (permalink / raw)
To: Ceph Development
[-- Attachment #1: Type: text/plain, Size: 817 bytes --]
Hi,
during the performance weely meeting, I had mentioned
my experiences concerning the transaction structure
for write requests at the level of the FileStore.
Such a transaction not only contains the OP_WRITE
operation to the object in the file system, but also
a series of OP_OMAP_SETKEYS and OP_SETATTR operations.
Find attached a README and source code patch, which
describe a prototype for coalescing the OP_OMAP_SETKEYS
operations and the performance impact f this change.
Regards
Andreas Bluemle
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
[-- Attachment #2: ceph-master.file-store-omap_setkeys-colaescing.patch --]
[-- Type: text/x-patch, Size: 1864 bytes --]
diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc
index f6c3bb8..29382b2 100644
--- a/src/os/FileStore.cc
+++ b/src/os/FileStore.cc
@@ -2260,10 +2260,24 @@ int FileStore::_check_replay_guard(int fd, const SequencerPosition& spos)
}
}
+void FileStore::_coalesce(map<string, bufferlist> &target, map<string, bufferlist> &source)
+{
+ for (map<string, bufferlist>::iterator p = source.begin();
+ p != source.end();
+ p++) {
+ target[p->first] = p->second;
+ }
+ return;
+}
+
unsigned FileStore::_do_transaction(
Transaction& t, uint64_t op_seq, int trans_num,
ThreadPool::TPHandle *handle)
{
+ map<string, bufferlist> collected_aset;
+ coll_t collected_cid;
+ ghobject_t collected_oid;
+
dout(10) << "_do_transaction on " << &t << dendl;
#ifdef WITH_LTTNG
@@ -2282,6 +2296,22 @@ unsigned FileStore::_do_transaction(
_inject_failure();
+ if (op->op == Transaction::OP_OMAP_SETKEYS) {
+ collected_cid = i.get_cid(op->cid);
+ collected_oid = i.get_oid(op->oid);
+ map<string, bufferlist> aset;
+ i.decode_attrset(aset);
+ _coalesce(collected_aset, aset);
+ continue;
+ } else {
+ if (collected_aset.empty() == false) {
+ tracepoint(objectstore, omap_setkeys_enter, osr_name);
+ r = _omap_setkeys(collected_cid, collected_oid, collected_aset, spos);
+ tracepoint(objectstore, omap_setkeys_exit, r);
+ collected_aset.clear();
+ }
+ }
+
switch (op->op) {
case Transaction::OP_NOP:
break;
diff --git a/src/os/FileStore.h b/src/os/FileStore.h
index af1fb8d..a039731 100644
--- a/src/os/FileStore.h
+++ b/src/os/FileStore.h
@@ -449,6 +449,8 @@ public:
int statfs(struct statfs *buf);
+ void _coalesce( map<string, bufferlist> &target, map<string, bufferlist> &source);
+
int _do_transactions(
list<Transaction*> &tls, uint64_t op_seq,
ThreadPool::TPHandle *handle);
[-- Attachment #3: README.file-store-coalescing --]
[-- Type: text/plain, Size: 2554 bytes --]
Coalescing OMAP_SETKEYS operations in a write transaction
---------------------------------------------------------
Description
-----------
At the level of FileStore, every write request is embedded in a transaction
which consists of
6 key-value pair settings in 3 OMAP_SETKEYS operations
the actual OP_WRITE
2 settings in the extended file system attributes.
The modification of the FileStore::_do_transaction() coalesces the
6 key-value pairs into a single operation, with the side effect of
reducing the number of key-value pairs to 5: one key appears twice
and only the last values is going to be set.
Performance improvement
-----------------------
Cluster with 3 storage nodes, 4 osd (SAS disk, SSD journal) per node,
separate client node with rbd using the kernel clients,
test load generated by fio, randon write, 4K block size, iodepth 16.
client improvement: approx. 5 % (12890 iops vs. 13369 iops)
storage node improvement: reduction in CPU consuptiom of ceph-osd daemon
by 10%; see follwoing table (derived from /proc/<pid>/schedstat:
ceph-osd process and CPU usage | CPU usage
thread classes v0.91 unmodified | v0.91 with coalescing
---------------------------------------------------+----------------------
total cpu usage: 43.17 CPU-seconds | 39.33 CPU-seconds
|
ThreadPool::WorkThread::entry(): 15.56 36.04% | 12.45 31.66%
ShardedThreadPool::workers: 8.07 18.70% | 7.94 20.18%
Pipe::Reader:: 5.81 13.45% | 5.92 15.04%
Pipe::Writer::entry(): 4.59 10.63% | 4.73 12.02%
FileJournal::Writer:: 2.41 5.57% | 2.45 6.22%
Finisher::finisher_thread: 2.86 6.63% | 1.03 2.61%
|
WBThrottle::entry: n/a n/a | 0.81 2.06%
Interesting: with coalescing active, the WBthrottle shows up in CPU usage.
In the default case, this was almost invisible.
Source/Patch
------------
https://www.github.com/andreas-bluemle/ceph
commit f33c48358f762cbeb5d30724efacf78ff5438e9e
patches:
relative to pull request at https://www.github.com/andreas-bluemle/ceph
ceph-andreas-bluemle.file-store-omap_setkeys-colaescing.patch
relative to ceph master at at https://www.github.com
(commit a7a70cabe25fdfe3322c784f6797231d14e112c2)
ceph-master.file-store-omap_setkeys-colaescing.patch
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-02-26 14:28 FileStore performance: coalescing operations Andreas Bluemle
@ 2015-02-26 15:02 ` Haomai Wang
2015-02-26 15:06 ` Mark Nelson
2015-03-04 1:05 ` Sage Weil
1 sibling, 1 reply; 9+ messages in thread
From: Haomai Wang @ 2015-02-26 15:02 UTC (permalink / raw)
To: Andreas Bluemle; +Cc: Ceph Development
Hmm, we already obverse this duplicate omap keys set from pglog operations.
And I think we need to resolve it in upper layer, of course,
coalescing omap operations in FileStore is also useful.
@Somnath Do you do this dedup work in KeyValueStore already?
On Thu, Feb 26, 2015 at 10:28 PM, Andreas Bluemle
<andreas.bluemle@itxperts.de> wrote:
> Hi,
>
> during the performance weely meeting, I had mentioned
> my experiences concerning the transaction structure
> for write requests at the level of the FileStore.
> Such a transaction not only contains the OP_WRITE
> operation to the object in the file system, but also
> a series of OP_OMAP_SETKEYS and OP_SETATTR operations.
>
> Find attached a README and source code patch, which
> describe a prototype for coalescing the OP_OMAP_SETKEYS
> operations and the performance impact f this change.
>
> Regards
>
> Andreas Bluemle
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-02-26 15:02 ` Haomai Wang
@ 2015-02-26 15:06 ` Mark Nelson
0 siblings, 0 replies; 9+ messages in thread
From: Mark Nelson @ 2015-02-26 15:06 UTC (permalink / raw)
To: Haomai Wang, Andreas Bluemle; +Cc: Ceph Development
On 02/26/2015 09:02 AM, Haomai Wang wrote:
> Hmm, we already obverse this duplicate omap keys set from pglog operations.
>
> And I think we need to resolve it in upper layer, of course,
Can we resolve this higher up easily? I am struck that it might be
easier to simply do the coalescing here. I think this was the approach
Sam was advocating in the performance meeting?
> coalescing omap operations in FileStore is also useful.
>
> @Somnath Do you do this dedup work in KeyValueStore already?
>
> On Thu, Feb 26, 2015 at 10:28 PM, Andreas Bluemle
> <andreas.bluemle@itxperts.de> wrote:
>> Hi,
>>
>> during the performance weely meeting, I had mentioned
>> my experiences concerning the transaction structure
>> for write requests at the level of the FileStore.
>> Such a transaction not only contains the OP_WRITE
>> operation to the object in the file system, but also
>> a series of OP_OMAP_SETKEYS and OP_SETATTR operations.
>>
>> Find attached a README and source code patch, which
>> describe a prototype for coalescing the OP_OMAP_SETKEYS
>> operations and the performance impact f this change.
>>
>> Regards
>>
>> Andreas Bluemle
>>
>> --
>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
>> ITXperts GmbH http://www.itxperts.de
>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>>
>> Company details: http://www.itxperts.de/imprint.htm
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-02-26 14:28 FileStore performance: coalescing operations Andreas Bluemle
2015-02-26 15:02 ` Haomai Wang
@ 2015-03-04 1:05 ` Sage Weil
2015-03-05 0:10 ` Sage Weil
1 sibling, 1 reply; 9+ messages in thread
From: Sage Weil @ 2015-03-04 1:05 UTC (permalink / raw)
To: Andreas Bluemle; +Cc: Ceph Development
On Thu, 26 Feb 2015, Andreas Bluemle wrote:
> Hi,
>
> during the performance weely meeting, I had mentioned
> my experiences concerning the transaction structure
> for write requests at the level of the FileStore.
> Such a transaction not only contains the OP_WRITE
> operation to the object in the file system, but also
> a series of OP_OMAP_SETKEYS and OP_SETATTR operations.
>
> Find attached a README and source code patch, which
> describe a prototype for coalescing the OP_OMAP_SETKEYS
> operations and the performance impact f this change.
I think we should try to avoid the dups in the first place in the upper
layers before resorting to deduping in the FileStore.
Here's a sample transaction:
{
"ops": [
{
"op_num": 0,
"op_name": "omap_setkeys",
"collection": "0.3_head",
"oid": "3\/\/head\/\/0",
"attr_lens": {
"0000000005.00000000000000000057": 180
}
},
^ PG::append_log
{
"op_num": 1,
"op_name": "omap_setkeys",
"collection": "0.3_head",
"oid": "3\/\/head\/\/0",
"attr_lens": {
"_epoch": 4,
"_info": 729
}
},
^^ these two come from PG::_write_info()
{
"op_num": 2,
"op_name": "omap_setkeys",
"collection": "0.3_head",
"oid": "3\/\/head\/\/0",
"attr_lens": {
"0000000005.00000000000000000057": 180,
"can_rollback_to": 12,
"rollback_info_trimmed_to": 12
}
},
^ PG::_write_log()
I think the log ones are easy to combine.. in fact I think we saw a pull
request from somone reently that does this?
The _write_info() one is probably a slightly bigger challenge, but only
slightly, given that both of these are called from
void PG::write_if_dirty(ObjectStore::Transaction& t)
{
if (dirty_big_info || dirty_info)
write_info(t);
pg_log.write_log(t, coll, pgmeta_oid);
}
{
"op_num": 3,
"op_name": "write",
"collection": "0.3_head",
"oid": "b2082803\/benchmark_data_maetl_20149_object474\/head\/\/0",
"length": 123,
"offset": 0,
"bufferlist length": 123
},
necessary
{
"op_num": 4,
"op_name": "setattr",
"collection": "0.3_head",
"oid": "b2082803\/benchmark_data_maetl_20149_object474\/head\/\/0",
"name": "_",
"length": 271
},
this too
{
"op_num": 5,
"op_name": "setattr",
"collection": "0.3_head",
"oid": "b2082803\/benchmark_data_maetl_20149_object474\/head\/\/0",
"name": "snapset",
"length": 31
}
We have a PR that avoids this second setattr when the value isn't
changing, but at the moment the consensus is that it's complex and
probably not worth merging. It's also a much cheaper operation, so I
suggest we fix the omap calls first and then compare performance w/ and
w/o the snapset xattr changes to see how significant it is.
]
}
Thanks!
sage
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-03-04 1:05 ` Sage Weil
@ 2015-03-05 0:10 ` Sage Weil
2015-03-05 7:04 ` Haomai Wang
0 siblings, 1 reply; 9+ messages in thread
From: Sage Weil @ 2015-03-05 0:10 UTC (permalink / raw)
To: Andreas Bluemle; +Cc: Ceph Development
On Tue, 3 Mar 2015, Sage Weil wrote:
> I think we should try to avoid the dups in the first place in the upper
> layers before resorting to deduping in the FileStore.
I took a stab at this:
https://github.com/ceph/ceph/pull/3878
Here's what the txns look like now:
{
"ops": [
{
"op_num": 0,
"op_name": "omap_setkeys",
"collection": "0.2_head",
"oid": "2\/\/head\/\/0",
"attr_lens": {
"0000000005.00000000000000000002": 178,
"_epoch": 4,
"_info": 729,
"can_rollback_to": 12,
"rollback_info_trimmed_to": 12
}
},
{
"op_num": 1,
"op_name": "write",
"collection": "0.2_head",
"oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
"length": 123,
"offset": 0,
"bufferlist length": 123
},
{
"op_num": 2,
"op_name": "setattr",
"collection": "0.2_head",
"oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
"name": "_",
"length": 269
},
{
"op_num": 3,
"op_name": "setattr",
"collection": "0.2_head",
"oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
"name": "snapset",
"length": 31
}
]
}
Care to give that a spin and see if it give syou a similar (or better)
improvement?
sage
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-03-05 0:10 ` Sage Weil
@ 2015-03-05 7:04 ` Haomai Wang
2015-03-11 3:44 ` Ning Yao
2015-03-19 14:59 ` Andreas Bluemle
0 siblings, 2 replies; 9+ messages in thread
From: Haomai Wang @ 2015-03-05 7:04 UTC (permalink / raw)
To: Sage Weil; +Cc: Andreas Bluemle, Ceph Development
I think the performance improvement can be refer to
https://github.com/ceph/ceph/pull/2972 which I did a simple benchmark
comparison.
This coalescing should get a lighter improvement but I think obviously
it should be better.
On Thu, Mar 5, 2015 at 8:10 AM, Sage Weil <sage@newdream.net> wrote:
> On Tue, 3 Mar 2015, Sage Weil wrote:
>> I think we should try to avoid the dups in the first place in the upper
>> layers before resorting to deduping in the FileStore.
>
> I took a stab at this:
>
> https://github.com/ceph/ceph/pull/3878
>
> Here's what the txns look like now:
>
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "omap_setkeys",
> "collection": "0.2_head",
> "oid": "2\/\/head\/\/0",
> "attr_lens": {
> "0000000005.00000000000000000002": 178,
> "_epoch": 4,
> "_info": 729,
> "can_rollback_to": 12,
> "rollback_info_trimmed_to": 12
> }
> },
> {
> "op_num": 1,
> "op_name": "write",
> "collection": "0.2_head",
> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> "length": 123,
> "offset": 0,
> "bufferlist length": 123
> },
> {
> "op_num": 2,
> "op_name": "setattr",
> "collection": "0.2_head",
> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> "name": "_",
> "length": 269
> },
> {
> "op_num": 3,
> "op_name": "setattr",
> "collection": "0.2_head",
> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> "name": "snapset",
> "length": 31
> }
> ]
> }
>
> Care to give that a spin and see if it give syou a similar (or better)
> improvement?
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-03-05 7:04 ` Haomai Wang
@ 2015-03-11 3:44 ` Ning Yao
2015-03-11 12:34 ` Sage Weil
2015-03-19 14:59 ` Andreas Bluemle
1 sibling, 1 reply; 9+ messages in thread
From: Ning Yao @ 2015-03-11 3:44 UTC (permalink / raw)
To: Haomai Wang; +Cc: Sage Weil, Andreas Bluemle, Ceph Development
Can we also consider to coalesce two OP_SETATTR transaction to a
single OP_SETATTRS transaction?
Regards
Ning Yao
2015-03-05 15:04 GMT+08:00 Haomai Wang <haomaiwang@gmail.com>:
> I think the performance improvement can be refer to
> https://github.com/ceph/ceph/pull/2972 which I did a simple benchmark
> comparison.
>
> This coalescing should get a lighter improvement but I think obviously
> it should be better.
>
> On Thu, Mar 5, 2015 at 8:10 AM, Sage Weil <sage@newdream.net> wrote:
>> On Tue, 3 Mar 2015, Sage Weil wrote:
>>> I think we should try to avoid the dups in the first place in the upper
>>> layers before resorting to deduping in the FileStore.
>>
>> I took a stab at this:
>>
>> https://github.com/ceph/ceph/pull/3878
>>
>> Here's what the txns look like now:
>>
>> {
>> "ops": [
>> {
>> "op_num": 0,
>> "op_name": "omap_setkeys",
>> "collection": "0.2_head",
>> "oid": "2\/\/head\/\/0",
>> "attr_lens": {
>> "0000000005.00000000000000000002": 178,
>> "_epoch": 4,
>> "_info": 729,
>> "can_rollback_to": 12,
>> "rollback_info_trimmed_to": 12
>> }
>> },
>> {
>> "op_num": 1,
>> "op_name": "write",
>> "collection": "0.2_head",
>> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
>> "length": 123,
>> "offset": 0,
>> "bufferlist length": 123
>> },
>> {
>> "op_num": 2,
>> "op_name": "setattr",
>> "collection": "0.2_head",
>> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
>> "name": "_",
>> "length": 269
>> },
>> {
>> "op_num": 3,
>> "op_name": "setattr",
>> "collection": "0.2_head",
>> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
>> "name": "snapset",
>> "length": 31
>> }
>> ]
>> }
>>
>> Care to give that a spin and see if it give syou a similar (or better)
>> improvement?
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards,
>
> Wheat
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-03-11 3:44 ` Ning Yao
@ 2015-03-11 12:34 ` Sage Weil
0 siblings, 0 replies; 9+ messages in thread
From: Sage Weil @ 2015-03-11 12:34 UTC (permalink / raw)
To: Ning Yao; +Cc: Haomai Wang, Andreas Bluemle, Ceph Development
On Wed, 11 Mar 2015, Ning Yao wrote:
> Can we also consider to coalesce two OP_SETATTR transaction to a
> single OP_SETATTRS transaction?
Yeah, definitely!
sage
> Regards
> Ning Yao
>
>
> 2015-03-05 15:04 GMT+08:00 Haomai Wang <haomaiwang@gmail.com>:
> > I think the performance improvement can be refer to
> > https://github.com/ceph/ceph/pull/2972 which I did a simple benchmark
> > comparison.
> >
> > This coalescing should get a lighter improvement but I think obviously
> > it should be better.
> >
> > On Thu, Mar 5, 2015 at 8:10 AM, Sage Weil <sage@newdream.net> wrote:
> >> On Tue, 3 Mar 2015, Sage Weil wrote:
> >>> I think we should try to avoid the dups in the first place in the upper
> >>> layers before resorting to deduping in the FileStore.
> >>
> >> I took a stab at this:
> >>
> >> https://github.com/ceph/ceph/pull/3878
> >>
> >> Here's what the txns look like now:
> >>
> >> {
> >> "ops": [
> >> {
> >> "op_num": 0,
> >> "op_name": "omap_setkeys",
> >> "collection": "0.2_head",
> >> "oid": "2\/\/head\/\/0",
> >> "attr_lens": {
> >> "0000000005.00000000000000000002": 178,
> >> "_epoch": 4,
> >> "_info": 729,
> >> "can_rollback_to": 12,
> >> "rollback_info_trimmed_to": 12
> >> }
> >> },
> >> {
> >> "op_num": 1,
> >> "op_name": "write",
> >> "collection": "0.2_head",
> >> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> >> "length": 123,
> >> "offset": 0,
> >> "bufferlist length": 123
> >> },
> >> {
> >> "op_num": 2,
> >> "op_name": "setattr",
> >> "collection": "0.2_head",
> >> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> >> "name": "_",
> >> "length": 269
> >> },
> >> {
> >> "op_num": 3,
> >> "op_name": "setattr",
> >> "collection": "0.2_head",
> >> "oid": "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> >> "name": "snapset",
> >> "length": 31
> >> }
> >> ]
> >> }
> >>
> >> Care to give that a spin and see if it give syou a similar (or better)
> >> improvement?
> >>
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> > --
> > Best Regards,
> >
> > Wheat
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: FileStore performance: coalescing operations
2015-03-05 7:04 ` Haomai Wang
2015-03-11 3:44 ` Ning Yao
@ 2015-03-19 14:59 ` Andreas Bluemle
1 sibling, 0 replies; 9+ messages in thread
From: Andreas Bluemle @ 2015-03-19 14:59 UTC (permalink / raw)
To: Sage Weil; +Cc: Ceph Development
Hi Sage,
as discussed yesterday during the performance meeting,
I have integratged pull reuest 3878 into v0.93 source code
and re-tested: I can confirm that the changes show a significant
benefit in the CPU consumption of the ceph-osd daemon for a
given workload: the reduction in CPU cosumption for the FileStore
workers is 22% and 10% for the overall CPU consumption of the
ceph-osd daemon.
Regards
Andreas Bluemle
On Thu, 5 Mar 2015 15:04:41 +0800
Haomai Wang <haomaiwang@gmail.com> wrote:
> I think the performance improvement can be refer to
> https://github.com/ceph/ceph/pull/2972 which I did a simple benchmark
> comparison.
>
> This coalescing should get a lighter improvement but I think obviously
> it should be better.
>
> On Thu, Mar 5, 2015 at 8:10 AM, Sage Weil <sage@newdream.net> wrote:
> > On Tue, 3 Mar 2015, Sage Weil wrote:
> >> I think we should try to avoid the dups in the first place in the
> >> upper layers before resorting to deduping in the FileStore.
> >
> > I took a stab at this:
> >
> > https://github.com/ceph/ceph/pull/3878
> >
> > Here's what the txns look like now:
> >
> > {
> > "ops": [
> > {
> > "op_num": 0,
> > "op_name": "omap_setkeys",
> > "collection": "0.2_head",
> > "oid": "2\/\/head\/\/0",
> > "attr_lens": {
> > "0000000005.00000000000000000002": 178,
> > "_epoch": 4,
> > "_info": 729,
> > "can_rollback_to": 12,
> > "rollback_info_trimmed_to": 12
> > }
> > },
> > {
> > "op_num": 1,
> > "op_name": "write",
> > "collection": "0.2_head",
> > "oid":
> > "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0",
> > "length": 123, "offset": 0,
> > "bufferlist length": 123
> > },
> > {
> > "op_num": 2,
> > "op_name": "setattr",
> > "collection": "0.2_head",
> > "oid":
> > "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0", "name":
> > "_", "length": 269
> > },
> > {
> > "op_num": 3,
> > "op_name": "setattr",
> > "collection": "0.2_head",
> > "oid":
> > "8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0", "name":
> > "snapset", "length": 31
> > }
> > ]
> > }
> >
> > Care to give that a spin and see if it give syou a similar (or
> > better) improvement?
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-03-19 14:59 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-02-26 14:28 FileStore performance: coalescing operations Andreas Bluemle
2015-02-26 15:02 ` Haomai Wang
2015-02-26 15:06 ` Mark Nelson
2015-03-04 1:05 ` Sage Weil
2015-03-05 0:10 ` Sage Weil
2015-03-05 7:04 ` Haomai Wang
2015-03-11 3:44 ` Ning Yao
2015-03-11 12:34 ` Sage Weil
2015-03-19 14:59 ` Andreas Bluemle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.