wip-auth

wip-auth

[Sage Weil]

Hi Andreas,

I took a look at the wip-auth I mentioned in the security call last 
week... and the patch didn't work at all.  Sorry if you wasted any 
time trying it.

Anyway, I fixed it up so that it actually worked and made one other 
optimization.  It would be great to hear what latencies you measure with 
the changes in place.

Also, it might be worth trying --with-cryptopp (or --with-nss if you built 
cryptopp by default) to see if there is a difference.  There is a ton of 
boilerplate setting up encryption contexts and key structures and so on 
that I suspect could be cached (perhaps stashed in the CryptoKey struct?) 
with a bit of effort.  See

	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213

sage

RE: wip-auth

[Blinick, Stephen L]

I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.    

Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7 
Kernel: 3.10.0-123.13.2.el7

100% 4K Writes, 1xOSD w/ Rados Bench
	libnss		            |    Cryptopp		
# QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
16	14432.57	1.11    |	18896.60	0.85	30.93%
	                                     
100% 4K Reads, 1xOSD w/ Rados Bench				
	libnss	                            |    Cryptopp		
# QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
16	19532.53	0.82    |	25708.70	0.62	31.62%


Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Thursday, January 22, 2015 4:56 PM
To: andreas.bluemle@itxperts.de
Cc: ceph-devel@vger.kernel.org
Subject: wip-auth

Hi Andreas,

I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.

Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.

Also, it might be worth trying --with-cryptopp (or --with-nss if you built cryptopp by default) to see if there is a difference.  There is a ton of boilerplate setting up encryption contexts and key structures and so on that I suspect could be cached (perhaps stashed in the CryptoKey struct?) with a bit of effort.  See

	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: wip-auth

[Sage Weil]

On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.    
> 
> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7 
> Kernel: 3.10.0-123.13.2.el7
> 
> 100% 4K Writes, 1xOSD w/ Rados Bench
> 	libnss		            |    Cryptopp		
> # QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
> 16	14432.57	1.11    |	18896.60	0.85	30.93%
> 	                                     
> 100% 4K Reads, 1xOSD w/ Rados Bench				
> 	libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms) IOPS 
> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%

Yikes, 30%!  I think this definitely worth some effort.  We switched to 
libnss because it has the weird government certfiications that everyone 
wants and is more prevalent.  crypto++ is also not packaged for Red 
Hat distros at all (presumably for that reason).

I suspect that most of the overhead is in the encryption context setup and 
can be avoided with a bit of effort..

sage


> 
> 
> Thanks,
> 
> Stephen
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Thursday, January 22, 2015 4:56 PM
> To: andreas.bluemle@itxperts.de
> Cc: ceph-devel@vger.kernel.org
> Subject: wip-auth
> 
> Hi Andreas,
> 
> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.
> 
> Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.
> 
> Also, it might be worth trying --with-cryptopp (or --with-nss if you built cryptopp by default) to see if there is a difference.  There is a ton of boilerplate setting up encryption contexts and key structures and so on that I suspect could be cached (perhaps stashed in the CryptoKey struct?) with a bit of effort.  See
> 
> 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: wip-auth

[Blinick, Stephen L]

Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked).

I forgot to mention the numbers below were based on v.91.

Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Monday, January 26, 2015 10:24 AM
To: Blinick, Stephen L
Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
Subject: RE: wip-auth

On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.    
> 
> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
> Kernel: 3.10.0-123.13.2.el7
> 
> 100% 4K Writes, 1xOSD w/ Rados Bench
> 	libnss		            |    Cryptopp		
> # QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
> 16	14432.57	1.11    |	18896.60	0.85	30.93%
> 	                                     
> 100% 4K Reads, 1xOSD w/ Rados Bench				
> 	libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms) IOPS 
> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%

Yikes, 30%!  I think this definitely worth some effort.  We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent.  crypto++ is also not packaged for Red Hat distros at all (presumably for that reason).

I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort..

sage


> 
> 
> Thanks,
> 
> Stephen
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Thursday, January 22, 2015 4:56 PM
> To: andreas.bluemle@itxperts.de
> Cc: ceph-devel@vger.kernel.org
> Subject: wip-auth
> 
> Hi Andreas,
> 
> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.
> 
> Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.
> 
> Also, it might be worth trying --with-cryptopp (or --with-nss if you 
> built cryptopp by default) to see if there is a difference.  There is 
> a ton of boilerplate setting up encryption contexts and key structures 
> and so on that I suspect could be cached (perhaps stashed in the 
> CryptoKey struct?) with a bit of effort.  See
> 
> 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: wip-auth

[Mark Nelson]

Hi Stephen,

Does this explain the results you were seeing earlier with the memstore 
testing?

Mark

On 01/26/2015 12:00 PM, Blinick, Stephen L wrote:
> Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked).
>
> I forgot to mention the numbers below were based on v.91.
>
> Thanks,
>
> Stephen
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Monday, January 26, 2015 10:24 AM
> To: Blinick, Stephen L
> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> Subject: RE: wip-auth
>
> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.
>>
>> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
>> Kernel: 3.10.0-123.13.2.el7
>>
>> 100% 4K Writes, 1xOSD w/ Rados Bench
>> 	libnss		            |    Cryptopp		
>> # QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
>> 16	14432.57	1.11    |	18896.60	0.85	30.93%
>> 	
>> 100% 4K Reads, 1xOSD w/ Rados Bench				
>> 	libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms) IOPS
>> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
>
> Yikes, 30%!  I think this definitely worth some effort.  We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent.  crypto++ is also not packaged for Red Hat distros at all (presumably for that reason).
>
> I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort..
>
> sage
>
>
>>
>>
>> Thanks,
>>
>> Stephen
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org
>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>> Sent: Thursday, January 22, 2015 4:56 PM
>> To: andreas.bluemle@itxperts.de
>> Cc: ceph-devel@vger.kernel.org
>> Subject: wip-auth
>>
>> Hi Andreas,
>>
>> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.
>>
>> Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.
>>
>> Also, it might be worth trying --with-cryptopp (or --with-nss if you
>> built cryptopp by default) to see if there is a difference.  There is
>> a ton of boilerplate setting up encryption contexts and key structures
>> and so on that I suspect could be cached (perhaps stashed in the
>> CryptoKey struct?) with a bit of effort.  See
>>
>> 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: wip-auth

[Blinick, Stephen L]

Hi Mark --  it doesn't, but it helps explain why there's some variance in all the various measurements.  I've only been running with the various debug settings off, but message signing, crc, authentication, etc at defaults.     I know some of the other results are with everything off, and that seems to have a large impact. 

Somewhere when switching versions, libnss became default, and so I was comparing RHEL7 w/ libnss to Ubuntu w/ Cryptopp.   When I switched to Cryptopp with RHEL7 below the numbers in my environment improved again. 

However, since libnss is the direction, I'll stick to that (and hopefully make back those improvements and more in the latest wip-auth commits).

Seems complex enough it's worth talking through verbally :)

Thanks,


Stephen


-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@inktank.com] 
Sent: Monday, January 26, 2015 11:16 AM
To: Blinick, Stephen L; Sage Weil
Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
Subject: Re: wip-auth

Hi Stephen,

Does this explain the results you were seeing earlier with the memstore testing?

Mark

On 01/26/2015 12:00 PM, Blinick, Stephen L wrote:
> Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked).
>
> I forgot to mention the numbers below were based on v.91.
>
> Thanks,
>
> Stephen
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Monday, January 26, 2015 10:24 AM
> To: Blinick, Stephen L
> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> Subject: RE: wip-auth
>
> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.
>>
>> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
>> Kernel: 3.10.0-123.13.2.el7
>>
>> 100% 4K Writes, 1xOSD w/ Rados Bench
>> 	libnss		            |    Cryptopp		
>> # QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
>> 16	14432.57	1.11    |	18896.60	0.85	30.93%
>> 	
>> 100% 4K Reads, 1xOSD w/ Rados Bench				
>> 	libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms) IOPS 
>> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
>
> Yikes, 30%!  I think this definitely worth some effort.  We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent.  crypto++ is also not packaged for Red Hat distros at all (presumably for that reason).
>
> I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort..
>
> sage
>
>
>>
>>
>> Thanks,
>>
>> Stephen
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org 
>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>> Sent: Thursday, January 22, 2015 4:56 PM
>> To: andreas.bluemle@itxperts.de
>> Cc: ceph-devel@vger.kernel.org
>> Subject: wip-auth
>>
>> Hi Andreas,
>>
>> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.
>>
>> Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.
>>
>> Also, it might be worth trying --with-cryptopp (or --with-nss if you 
>> built cryptopp by default) to see if there is a difference.  There is 
>> a ton of boilerplate setting up encryption contexts and key 
>> structures and so on that I suspect could be cached (perhaps stashed 
>> in the CryptoKey struct?) with a bit of effort.  See
>>
>> 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>

RE: wip-auth

[Sage Weil]

I spent some time focusing on just CryptoKey::encrypt().  I benchmarked 
100,000 encrypts of 128 bytes and got, at baseline,

 cryptopp: 100000 encoded in 0.655651
 libnss  : 100000 encoded in 1.288786

Ouch!  With a (fixed) version of my earlier patch that avoids uselessly 
copying the input buffer:

 100000 encoded in 1.231977

With a patch that puts the key structures in CryptoKey instead of 
recreating them each time:

 100000 encoded in 0.396208  -- ~70% improvement over original

This is pushed to wip-auth.  There's also a patch that caches key structs 
for crypopp.. it now takes

 100000 encoded in 0.440758  -- ~33% improvement over original

(Not that almost anybody will ever care; we use libnss by default for both 
rpm and deb distros.)

So, yay, nss is now a bit faster.  What I'm not completely certain about 
is whether the structures I've preserved are truly stateless (and can be 
shared across threads, etc.).  They encrypt/decrypt methods are const so, 
if the libraries are const-correct, it should be fine... but perhaps 
someone familiar with nss and/or crypto++ can review this?

This is pushed to the latest wip-auth branch:

	https://github.com/ceph/ceph/commits/wip-auth

Andreas and Stephen, what effect does this have on your numbers?

Thanks!
sage


On Mon, 26 Jan 2015, Blinick, Stephen L wrote:

> Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked).
> 
> I forgot to mention the numbers below were based on v.91.
> 
> Thanks,
> 
> Stephen
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Monday, January 26, 2015 10:24 AM
> To: Blinick, Stephen L
> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> Subject: RE: wip-auth
> 
> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> > I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++.  Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client.    
> > 
> > Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
> > Kernel: 3.10.0-123.13.2.el7
> > 
> > 100% 4K Writes, 1xOSD w/ Rados Bench
> > 	libnss		            |    Cryptopp		
> > # QD	IOPS	Latency(ms)   |	IOPS	Latency(ms)	IOPS Improvement %
> > 16	14432.57	1.11    |	18896.60	0.85	30.93%
> > 	                                     
> > 100% 4K Reads, 1xOSD w/ Rados Bench				
> > 	libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms) IOPS 
> > Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
> 
> Yikes, 30%!  I think this definitely worth some effort.  We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent.  crypto++ is also not packaged for Red Hat distros at all (presumably for that reason).
> 
> I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort..
> 
> sage
> 
> 
> > 
> > 
> > Thanks,
> > 
> > Stephen
> > 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org 
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Thursday, January 22, 2015 4:56 PM
> > To: andreas.bluemle@itxperts.de
> > Cc: ceph-devel@vger.kernel.org
> > Subject: wip-auth
> > 
> > Hi Andreas,
> > 
> > I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all.  Sorry if you wasted any time trying it.
> > 
> > Anyway, I fixed it up so that it actually worked and made one other optimization.  It would be great to hear what latencies you measure with the changes in place.
> > 
> > Also, it might be worth trying --with-cryptopp (or --with-nss if you 
> > built cryptopp by default) to see if there is a difference.  There is 
> > a ton of boilerplate setting up encryption contexts and key structures 
> > and so on that I suspect could be cached (perhaps stashed in the 
> > CryptoKey struct?) with a bit of effort.  See
> > 
> > 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
> > 
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: wip-auth

[Andreas Bluemle]

[-- Attachment #1: Type: text/plain, Size: 6487 bytes --]

Hi Sage,

I tried to integrate wip-auth into my 0.91 build environment.

I had not been able to start the cluster successfully: ceph-mon
crashes with a segmentation fault in CryptoKey::encrypt()
(see attachment).

This happens when linking with libnss or libcryptopp (version 5.6.2).

I created the patch to add wip-auth based on github
pull request 3523 and was able to use this patch directly 
with 0.91 with only a minor adaptation for common/ceph_context.h:
the 0.91 version of ceph_context.h did not know anything about
the experimental "class CephContextObs".

wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.

As my build environment is rpm, I had to modify the invocation
of the configure script in the spec file instead of the do_autogen.sh
script.


Best Regards

Andreas Bluemle


On Tue, 27 Jan 2015 09:18:45 -0800 (PST)
Sage Weil <sweil@redhat.com> wrote:

> I spent some time focusing on just CryptoKey::encrypt().  I
> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
> 
>  cryptopp: 100000 encoded in 0.655651
>  libnss  : 100000 encoded in 1.288786
> 
> Ouch!  With a (fixed) version of my earlier patch that avoids
> uselessly copying the input buffer:
> 
>  100000 encoded in 1.231977
> 
> With a patch that puts the key structures in CryptoKey instead of 
> recreating them each time:
> 
>  100000 encoded in 0.396208  -- ~70% improvement over original
> 
> This is pushed to wip-auth.  There's also a patch that caches key
> structs for crypopp.. it now takes
> 
>  100000 encoded in 0.440758  -- ~33% improvement over original
> 
> (Not that almost anybody will ever care; we use libnss by default for
> both rpm and deb distros.)
> 
> So, yay, nss is now a bit faster.  What I'm not completely certain
> about is whether the structures I've preserved are truly stateless
> (and can be shared across threads, etc.).  They encrypt/decrypt
> methods are const so, if the libraries are const-correct, it should
> be fine... but perhaps someone familiar with nss and/or crypto++ can
> review this?
> 
> This is pushed to the latest wip-auth branch:
> 
> 	https://github.com/ceph/ceph/commits/wip-auth
> 
> Andreas and Stephen, what effect does this have on your numbers?
> 
> Thanks!
> sage
> 
> 
> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> 
> > Good to know, I was wondering why the spec file defaulted to
> > lib-nss.. the dpkg-build for debian packages just uses whatever
> > configuration you had built, and I believe that will use
> > libcryptopp if the dependency is installed on the build machine
> > (last I looked).
> > 
> > I forgot to mention the numbers below were based on v.91.
> > 
> > Thanks,
> > 
> > Stephen
> > 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
> > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> > Subject: RE: wip-auth
> > 
> > On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> > > I noticed that the spec file for building RPM's defaults to
> > > building with libnss, instead of libcrypto++.  Since the
> > > measurements I'd done so far were from those RPM's I rebuilt with
> > > libcrypto++.. so FWIW here is the difference between those two on
> > > my system, memstore backend with a single OSD, and single
> > > client.    
> > > 
> > > Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
> > > Kernel: 3.10.0-123.13.2.el7
> > > 
> > > 100% 4K Writes, 1xOSD w/ Rados Bench
> > > 	libnss		            |
> > > Cryptopp # QD	IOPS	Latency(ms)   |
> > > IOPS	Latency(ms)	IOPS Improvement % 16
> > > 14432.57	1.11    |	18896.60	0.85
> > > 30.93% 100% 4K Reads, 1xOSD w/ Rados
> > > Bench libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms)
> > > IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
> > 
> > Yikes, 30%!  I think this definitely worth some effort.  We
> > switched to libnss because it has the weird government
> > certfiications that everyone wants and is more prevalent.  crypto++
> > is also not packaged for Red Hat distros at all (presumably for
> > that reason).
> > 
> > I suspect that most of the overhead is in the encryption context
> > setup and can be avoided with a bit of effort..
> > 
> > sage
> > 
> > 
> > > 
> > > 
> > > Thanks,
> > > 
> > > Stephen
> > > 
> > > -----Original Message-----
> > > From: ceph-devel-owner@vger.kernel.org 
> > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > > Sent: Thursday, January 22, 2015 4:56 PM
> > > To: andreas.bluemle@itxperts.de
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: wip-auth
> > > 
> > > Hi Andreas,
> > > 
> > > I took a look at the wip-auth I mentioned in the security call
> > > last week... and the patch didn't work at all.  Sorry if you
> > > wasted any time trying it.
> > > 
> > > Anyway, I fixed it up so that it actually worked and made one
> > > other optimization.  It would be great to hear what latencies you
> > > measure with the changes in place.
> > > 
> > > Also, it might be worth trying --with-cryptopp (or --with-nss if
> > > you built cryptopp by default) to see if there is a difference.
> > > There is a ton of boilerplate setting up encryption contexts and
> > > key structures and so on that I suspect could be cached (perhaps
> > > stashed in the CryptoKey struct?) with a bit of effort.  See
> > > 
> > > 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
> > > 
> > > sage
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > -- To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

[-- Attachment #2: ceph-mon-0.91-wip-auth-backtrace.txt --]
[-- Type: text/plain, Size: 1093 bytes --]

0> 2015-01-30 08:00:15.644464 7f3229507700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f3229507700

 ceph version 0.91 (725d66098c98c2008b5fa07538325cc6816ca4a1)
 1: /usr/bin/ceph-mon() [0x8ea1a5]
 2: /lib64/libc.so.6() [0x316e4329a0]
 3: (CryptoKey::encrypt(CephContext*, ceph::buffer::list const&, ceph::buffer::list&, std::string&) const+0x41) [0x6c2c51]
 4: (void encode_encrypt_enc_bl<CephXServiceTicketInfo>(CephContext*, CephXServiceTicketInfo const&, CryptoKey const&, ceph::buffer::list&, std::string&)+0x26a) [0x6bf8ea]
 5: (cephx_build_service_ticket_blob(CephContext*, CephXSessionAuthInfo&, CephXTicketBlob&)+0x2b2) [0x6bb002]
 6: (Monitor::ms_get_authorizer(int, AuthAuthorizer**, bool)+0x39d) [0x57755d]
 7: (SimpleMessenger::get_authorizer(int, bool)+0x4b) [0x8b3e5b]
 8: (Pipe::connect()+0x1975) [0x8d8625]
 9: (Pipe::writer()+0x9f3) [0x8db713]
 10: (Pipe::Writer::entry()+0xd) [0x8de4ed]
 11: /lib64/libpthread.so.0() [0x316ec079d1]
 12: (clone()+0x6d) [0x316e4e8b7d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Re: wip-auth

[Sage Weil]

Hi Andreas,

It looks like that was a stale sha1, but the newer one was also 
broken.  I've retested and it's working for me now.  See latest wip-auth, 
sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.

Thanks!
sage

On Fri, 30 Jan 2015, Andreas Bluemle wrote:

> Hi Sage,
> 
> I tried to integrate wip-auth into my 0.91 build environment.
> 
> I had not been able to start the cluster successfully: ceph-mon
> crashes with a segmentation fault in CryptoKey::encrypt()
> (see attachment).
> 
> This happens when linking with libnss or libcryptopp (version 5.6.2).
> 
> I created the patch to add wip-auth based on github
> pull request 3523 and was able to use this patch directly 
> with 0.91 with only a minor adaptation for common/ceph_context.h:
> the 0.91 version of ceph_context.h did not know anything about
> the experimental "class CephContextObs".
> 
> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
> 
> As my build environment is rpm, I had to modify the invocation
> of the configure script in the spec file instead of the do_autogen.sh
> script.
> 
> 
> Best Regards
> 
> Andreas Bluemle
> 
> 
> On Tue, 27 Jan 2015 09:18:45 -0800 (PST)
> Sage Weil <sweil@redhat.com> wrote:
> 
> > I spent some time focusing on just CryptoKey::encrypt().  I
> > benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
> > 
> >  cryptopp: 100000 encoded in 0.655651
> >  libnss  : 100000 encoded in 1.288786
> > 
> > Ouch!  With a (fixed) version of my earlier patch that avoids
> > uselessly copying the input buffer:
> > 
> >  100000 encoded in 1.231977
> > 
> > With a patch that puts the key structures in CryptoKey instead of 
> > recreating them each time:
> > 
> >  100000 encoded in 0.396208  -- ~70% improvement over original
> > 
> > This is pushed to wip-auth.  There's also a patch that caches key
> > structs for crypopp.. it now takes
> > 
> >  100000 encoded in 0.440758  -- ~33% improvement over original
> > 
> > (Not that almost anybody will ever care; we use libnss by default for
> > both rpm and deb distros.)
> > 
> > So, yay, nss is now a bit faster.  What I'm not completely certain
> > about is whether the structures I've preserved are truly stateless
> > (and can be shared across threads, etc.).  They encrypt/decrypt
> > methods are const so, if the libraries are const-correct, it should
> > be fine... but perhaps someone familiar with nss and/or crypto++ can
> > review this?
> > 
> > This is pushed to the latest wip-auth branch:
> > 
> > 	https://github.com/ceph/ceph/commits/wip-auth
> > 
> > Andreas and Stephen, what effect does this have on your numbers?
> > 
> > Thanks!
> > sage
> > 
> > 
> > On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> > 
> > > Good to know, I was wondering why the spec file defaulted to
> > > lib-nss.. the dpkg-build for debian packages just uses whatever
> > > configuration you had built, and I believe that will use
> > > libcryptopp if the dependency is installed on the build machine
> > > (last I looked).
> > > 
> > > I forgot to mention the numbers below were based on v.91.
> > > 
> > > Thanks,
> > > 
> > > Stephen
> > > 
> > > -----Original Message-----
> > > From: ceph-devel-owner@vger.kernel.org
> > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > > Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
> > > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> > > Subject: RE: wip-auth
> > > 
> > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> > > > I noticed that the spec file for building RPM's defaults to
> > > > building with libnss, instead of libcrypto++.  Since the
> > > > measurements I'd done so far were from those RPM's I rebuilt with
> > > > libcrypto++.. so FWIW here is the difference between those two on
> > > > my system, memstore backend with a single OSD, and single
> > > > client.    
> > > > 
> > > > Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
> > > > Kernel: 3.10.0-123.13.2.el7
> > > > 
> > > > 100% 4K Writes, 1xOSD w/ Rados Bench
> > > > 	libnss		            |
> > > > Cryptopp # QD	IOPS	Latency(ms)   |
> > > > IOPS	Latency(ms)	IOPS Improvement % 16
> > > > 14432.57	1.11    |	18896.60	0.85
> > > > 30.93% 100% 4K Reads, 1xOSD w/ Rados
> > > > Bench libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms)
> > > > IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
> > > 
> > > Yikes, 30%!  I think this definitely worth some effort.  We
> > > switched to libnss because it has the weird government
> > > certfiications that everyone wants and is more prevalent.  crypto++
> > > is also not packaged for Red Hat distros at all (presumably for
> > > that reason).
> > > 
> > > I suspect that most of the overhead is in the encryption context
> > > setup and can be avoided with a bit of effort..
> > > 
> > > sage
> > > 
> > > 
> > > > 
> > > > 
> > > > Thanks,
> > > > 
> > > > Stephen
> > > > 
> > > > -----Original Message-----
> > > > From: ceph-devel-owner@vger.kernel.org 
> > > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > > > Sent: Thursday, January 22, 2015 4:56 PM
> > > > To: andreas.bluemle@itxperts.de
> > > > Cc: ceph-devel@vger.kernel.org
> > > > Subject: wip-auth
> > > > 
> > > > Hi Andreas,
> > > > 
> > > > I took a look at the wip-auth I mentioned in the security call
> > > > last week... and the patch didn't work at all.  Sorry if you
> > > > wasted any time trying it.
> > > > 
> > > > Anyway, I fixed it up so that it actually worked and made one
> > > > other optimization.  It would be great to hear what latencies you
> > > > measure with the changes in place.
> > > > 
> > > > Also, it might be worth trying --with-cryptopp (or --with-nss if
> > > > you built cryptopp by default) to see if there is a difference.
> > > > There is a ton of boilerplate setting up encryption contexts and
> > > > key structures and so on that I suspect could be cached (perhaps
> > > > stashed in the CryptoKey struct?) with a bit of effort.  See
> > > > 
> > > > 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
> > > > 
> > > > sage
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe
> > > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > -- To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > 
> > 
> > 
> 
> 
> 
> -- 
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> 
> Company details: http://www.itxperts.de/imprint.htm
> 

Re: wip-auth

[Mark Nelson]

Hi All,

I completed some tests with wip-auth to the memstore on ubuntu earlier 
today.  Basic gist of it is that the improvements in wip-auth help but 
don't quite get us to what can be achieved with auth disabled.  RHEL7 
(without auth) continues to do very well in latency bound situations. 
Next up will be to see if how much this matters when testing against on 
SSDs.

Here are the results:

sync 4k object writes
=====================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	0.99		1007
Wip-auth	Ubuntu 14.04	Yes	0.81		1237
master		Ubuntu 14.04	No	0.64		1549
Wip-auth	Ubuntu 14.04	No	0.65		1563
master		RHEL7		No	0.32		3158

sync 4k object reads
====================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	0.59		1695
Wip-auth	Ubuntu 14.04	Yes	0.41		2409
master		Ubuntu 14.04	No	0.29		3425
Wip-auth	Ubuntu 14.04	No	0.29		3474
master		RHEL7		No	0.17		5853

256 concurrent 4k object writes
===============================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	40.39		6339
Wip-auth	Ubuntu 14.04	Yes	26.22		9763
master		Ubuntu 14.04	No	17.46		14662
Wip-auth	Ubuntu 14.04	No	17.34		14759
master		RHEL7		No	14.93		17139

256 concurrent 4k object reads
==============================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	31.47		8134
Wip-auth	Ubuntu 14.04	Yes	19.81		12922
master		Ubuntu 14.04	No	12.82		19968
Wip-auth	Ubuntu 14.04	No	12.75		20080
master		RHEL7		No	12.04		21257

Mark

On 01/30/2015 03:08 PM, Sage Weil wrote:
> Hi Andreas,
>
> It looks like that was a stale sha1, but the newer one was also
> broken.  I've retested and it's working for me now.  See latest wip-auth,
> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.
>
> Thanks!
> sage
>
> On Fri, 30 Jan 2015, Andreas Bluemle wrote:
>
>> Hi Sage,
>>
>> I tried to integrate wip-auth into my 0.91 build environment.
>>
>> I had not been able to start the cluster successfully: ceph-mon
>> crashes with a segmentation fault in CryptoKey::encrypt()
>> (see attachment).
>>
>> This happens when linking with libnss or libcryptopp (version 5.6.2).
>>
>> I created the patch to add wip-auth based on github
>> pull request 3523 and was able to use this patch directly
>> with 0.91 with only a minor adaptation for common/ceph_context.h:
>> the 0.91 version of ceph_context.h did not know anything about
>> the experimental "class CephContextObs".
>>
>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
>>
>> As my build environment is rpm, I had to modify the invocation
>> of the configure script in the spec file instead of the do_autogen.sh
>> script.
>>
>>
>> Best Regards
>>
>> Andreas Bluemle
>>
>>
>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST)
>> Sage Weil <sweil@redhat.com> wrote:
>>
>>> I spent some time focusing on just CryptoKey::encrypt().  I
>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
>>>
>>>   cryptopp: 100000 encoded in 0.655651
>>>   libnss  : 100000 encoded in 1.288786
>>>
>>> Ouch!  With a (fixed) version of my earlier patch that avoids
>>> uselessly copying the input buffer:
>>>
>>>   100000 encoded in 1.231977
>>>
>>> With a patch that puts the key structures in CryptoKey instead of
>>> recreating them each time:
>>>
>>>   100000 encoded in 0.396208  -- ~70% improvement over original
>>>
>>> This is pushed to wip-auth.  There's also a patch that caches key
>>> structs for crypopp.. it now takes
>>>
>>>   100000 encoded in 0.440758  -- ~33% improvement over original
>>>
>>> (Not that almost anybody will ever care; we use libnss by default for
>>> both rpm and deb distros.)
>>>
>>> So, yay, nss is now a bit faster.  What I'm not completely certain
>>> about is whether the structures I've preserved are truly stateless
>>> (and can be shared across threads, etc.).  They encrypt/decrypt
>>> methods are const so, if the libraries are const-correct, it should
>>> be fine... but perhaps someone familiar with nss and/or crypto++ can
>>> review this?
>>>
>>> This is pushed to the latest wip-auth branch:
>>>
>>> 	https://github.com/ceph/ceph/commits/wip-auth
>>>
>>> Andreas and Stephen, what effect does this have on your numbers?
>>>
>>> Thanks!
>>> sage
>>>
>>>
>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>
>>>> Good to know, I was wondering why the spec file defaulted to
>>>> lib-nss.. the dpkg-build for debian packages just uses whatever
>>>> configuration you had built, and I believe that will use
>>>> libcryptopp if the dependency is installed on the build machine
>>>> (last I looked).
>>>>
>>>> I forgot to mention the numbers below were based on v.91.
>>>>
>>>> Thanks,
>>>>
>>>> Stephen
>>>>
>>>> -----Original Message-----
>>>> From: ceph-devel-owner@vger.kernel.org
>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
>>>> Subject: RE: wip-auth
>>>>
>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>>> I noticed that the spec file for building RPM's defaults to
>>>>> building with libnss, instead of libcrypto++.  Since the
>>>>> measurements I'd done so far were from those RPM's I rebuilt with
>>>>> libcrypto++.. so FWIW here is the difference between those two on
>>>>> my system, memstore backend with a single OSD, and single
>>>>> client.
>>>>>
>>>>> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
>>>>> Kernel: 3.10.0-123.13.2.el7
>>>>>
>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench
>>>>> 	libnss		            |
>>>>> Cryptopp # QD	IOPS	Latency(ms)   |
>>>>> IOPS	Latency(ms)	IOPS Improvement % 16
>>>>> 14432.57	1.11    |	18896.60	0.85
>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados
>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms)  | IOPS Latency(ms)
>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
>>>>
>>>> Yikes, 30%!  I think this definitely worth some effort.  We
>>>> switched to libnss because it has the weird government
>>>> certfiications that everyone wants and is more prevalent.  crypto++
>>>> is also not packaged for Red Hat distros at all (presumably for
>>>> that reason).
>>>>
>>>> I suspect that most of the overhead is in the encryption context
>>>> setup and can be avoided with a bit of effort..
>>>>
>>>> sage
>>>>
>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Stephen
>>>>>
>>>>> -----Original Message-----
>>>>> From: ceph-devel-owner@vger.kernel.org
>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>>> Sent: Thursday, January 22, 2015 4:56 PM
>>>>> To: andreas.bluemle@itxperts.de
>>>>> Cc: ceph-devel@vger.kernel.org
>>>>> Subject: wip-auth
>>>>>
>>>>> Hi Andreas,
>>>>>
>>>>> I took a look at the wip-auth I mentioned in the security call
>>>>> last week... and the patch didn't work at all.  Sorry if you
>>>>> wasted any time trying it.
>>>>>
>>>>> Anyway, I fixed it up so that it actually worked and made one
>>>>> other optimization.  It would be great to hear what latencies you
>>>>> measure with the changes in place.
>>>>>
>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if
>>>>> you built cryptopp by default) to see if there is a difference.
>>>>> There is a ton of boilerplate setting up encryption contexts and
>>>>> key structures and so on that I suspect could be cached (perhaps
>>>>> stashed in the CryptoKey struct?) with a bit of effort.  See
>>>>>
>>>>> 	https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
>>>>>
>>>>> sage
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> -- To unsubscribe from this list: send the line "unsubscribe
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
>> ITXperts GmbH                       http://www.itxperts.de
>> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
>> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>>
>> Company details: http://www.itxperts.de/imprint.htm
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: wip-auth

[Alexandre DERUMIER]

Hi Mark,

do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge)


Not sure it's related, but I have found a bug report on ubuntu,
about irqbalance not working correctly on ubuntu 14.04
https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065


----- Mail original -----
De: "Mark Nelson" <mark.nelson@inktank.com>
À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de>
Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Jeudi 5 Février 2015 00:24:42
Objet: Re: wip-auth

Hi All, 

I completed some tests with wip-auth to the memstore on ubuntu earlier 
today. Basic gist of it is that the improvements in wip-auth help but 
don't quite get us to what can be achieved with auth disabled. RHEL7 
(without auth) continues to do very well in latency bound situations. 
Next up will be to see if how much this matters when testing against on 
SSDs. 

Here are the results: 

sync 4k object writes 
===================== 

Ceph OS Auth Avg Lat (ms) Avg IOPS 
---------------------------------------------------------------- 
master Ubuntu 14.04 Yes 0.99 1007 
Wip-auth Ubuntu 14.04 Yes 0.81 1237 
master Ubuntu 14.04 No 0.64 1549 
Wip-auth Ubuntu 14.04 No 0.65 1563 
master RHEL7 No 0.32 3158 

sync 4k object reads 
==================== 

Ceph OS Auth Avg Lat (ms) Avg IOPS 
---------------------------------------------------------------- 
master Ubuntu 14.04 Yes 0.59 1695 
Wip-auth Ubuntu 14.04 Yes 0.41 2409 
master Ubuntu 14.04 No 0.29 3425 
Wip-auth Ubuntu 14.04 No 0.29 3474 
master RHEL7 No 0.17 5853 

256 concurrent 4k object writes 
=============================== 

Ceph OS Auth Avg Lat (ms) Avg IOPS 
---------------------------------------------------------------- 
master Ubuntu 14.04 Yes 40.39 6339 
Wip-auth Ubuntu 14.04 Yes 26.22 9763 
master Ubuntu 14.04 No 17.46 14662 
Wip-auth Ubuntu 14.04 No 17.34 14759 
master RHEL7 No 14.93 17139 

256 concurrent 4k object reads 
============================== 

Ceph OS Auth Avg Lat (ms) Avg IOPS 
---------------------------------------------------------------- 
master Ubuntu 14.04 Yes 31.47 8134 
Wip-auth Ubuntu 14.04 Yes 19.81 12922 
master Ubuntu 14.04 No 12.82 19968 
Wip-auth Ubuntu 14.04 No 12.75 20080 
master RHEL7 No 12.04 21257 

Mark 

On 01/30/2015 03:08 PM, Sage Weil wrote: 
> Hi Andreas, 
> 
> It looks like that was a stale sha1, but the newer one was also 
> broken. I've retested and it's working for me now. See latest wip-auth, 
> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. 
> 
> Thanks! 
> sage 
> 
> On Fri, 30 Jan 2015, Andreas Bluemle wrote: 
> 
>> Hi Sage, 
>> 
>> I tried to integrate wip-auth into my 0.91 build environment. 
>> 
>> I had not been able to start the cluster successfully: ceph-mon 
>> crashes with a segmentation fault in CryptoKey::encrypt() 
>> (see attachment). 
>> 
>> This happens when linking with libnss or libcryptopp (version 5.6.2). 
>> 
>> I created the patch to add wip-auth based on github 
>> pull request 3523 and was able to use this patch directly 
>> with 0.91 with only a minor adaptation for common/ceph_context.h: 
>> the 0.91 version of ceph_context.h did not know anything about 
>> the experimental "class CephContextObs". 
>> 
>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. 
>> 
>> As my build environment is rpm, I had to modify the invocation 
>> of the configure script in the spec file instead of the do_autogen.sh 
>> script. 
>> 
>> 
>> Best Regards 
>> 
>> Andreas Bluemle 
>> 
>> 
>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) 
>> Sage Weil <sweil@redhat.com> wrote: 
>> 
>>> I spent some time focusing on just CryptoKey::encrypt(). I 
>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, 
>>> 
>>> cryptopp: 100000 encoded in 0.655651 
>>> libnss : 100000 encoded in 1.288786 
>>> 
>>> Ouch! With a (fixed) version of my earlier patch that avoids 
>>> uselessly copying the input buffer: 
>>> 
>>> 100000 encoded in 1.231977 
>>> 
>>> With a patch that puts the key structures in CryptoKey instead of 
>>> recreating them each time: 
>>> 
>>> 100000 encoded in 0.396208 -- ~70% improvement over original 
>>> 
>>> This is pushed to wip-auth. There's also a patch that caches key 
>>> structs for crypopp.. it now takes 
>>> 
>>> 100000 encoded in 0.440758 -- ~33% improvement over original 
>>> 
>>> (Not that almost anybody will ever care; we use libnss by default for 
>>> both rpm and deb distros.) 
>>> 
>>> So, yay, nss is now a bit faster. What I'm not completely certain 
>>> about is whether the structures I've preserved are truly stateless 
>>> (and can be shared across threads, etc.). They encrypt/decrypt 
>>> methods are const so, if the libraries are const-correct, it should 
>>> be fine... but perhaps someone familiar with nss and/or crypto++ can 
>>> review this? 
>>> 
>>> This is pushed to the latest wip-auth branch: 
>>> 
>>> https://github.com/ceph/ceph/commits/wip-auth 
>>> 
>>> Andreas and Stephen, what effect does this have on your numbers? 
>>> 
>>> Thanks! 
>>> sage 
>>> 
>>> 
>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: 
>>> 
>>>> Good to know, I was wondering why the spec file defaulted to 
>>>> lib-nss.. the dpkg-build for debian packages just uses whatever 
>>>> configuration you had built, and I believe that will use 
>>>> libcryptopp if the dependency is installed on the build machine 
>>>> (last I looked). 
>>>> 
>>>> I forgot to mention the numbers below were based on v.91. 
>>>> 
>>>> Thanks, 
>>>> 
>>>> Stephen 
>>>> 
>>>> -----Original Message----- 
>>>> From: ceph-devel-owner@vger.kernel.org 
>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil 
>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L 
>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org 
>>>> Subject: RE: wip-auth 
>>>> 
>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: 
>>>>> I noticed that the spec file for building RPM's defaults to 
>>>>> building with libnss, instead of libcrypto++. Since the 
>>>>> measurements I'd done so far were from those RPM's I rebuilt with 
>>>>> libcrypto++.. so FWIW here is the difference between those two on 
>>>>> my system, memstore backend with a single OSD, and single 
>>>>> client. 
>>>>> 
>>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 
>>>>> Kernel: 3.10.0-123.13.2.el7 
>>>>> 
>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench 
>>>>> libnss | 
>>>>> Cryptopp # QD IOPS Latency(ms) | 
>>>>> IOPS Latency(ms) IOPS Improvement % 16 
>>>>> 14432.57 1.11 | 18896.60 0.85 
>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados 
>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) 
>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% 
>>>> 
>>>> Yikes, 30%! I think this definitely worth some effort. We 
>>>> switched to libnss because it has the weird government 
>>>> certfiications that everyone wants and is more prevalent. crypto++ 
>>>> is also not packaged for Red Hat distros at all (presumably for 
>>>> that reason). 
>>>> 
>>>> I suspect that most of the overhead is in the encryption context 
>>>> setup and can be avoided with a bit of effort.. 
>>>> 
>>>> sage 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> Thanks, 
>>>>> 
>>>>> Stephen 
>>>>> 
>>>>> -----Original Message----- 
>>>>> From: ceph-devel-owner@vger.kernel.org 
>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil 
>>>>> Sent: Thursday, January 22, 2015 4:56 PM 
>>>>> To: andreas.bluemle@itxperts.de 
>>>>> Cc: ceph-devel@vger.kernel.org 
>>>>> Subject: wip-auth 
>>>>> 
>>>>> Hi Andreas, 
>>>>> 
>>>>> I took a look at the wip-auth I mentioned in the security call 
>>>>> last week... and the patch didn't work at all. Sorry if you 
>>>>> wasted any time trying it. 
>>>>> 
>>>>> Anyway, I fixed it up so that it actually worked and made one 
>>>>> other optimization. It would be great to hear what latencies you 
>>>>> measure with the changes in place. 
>>>>> 
>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if 
>>>>> you built cryptopp by default) to see if there is a difference. 
>>>>> There is a ton of boilerplate setting up encryption contexts and 
>>>>> key structures and so on that I suspect could be cached (perhaps 
>>>>> stashed in the CryptoKey struct?) with a bit of effort. See 
>>>>> 
>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 
>>>>> 
>>>>> sage 
>>>>> -- 
>>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>>> 
>>>>> 
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>> -- To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de 
>> ITXperts GmbH http://www.itxperts.de 
>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 
>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 
>> 
>> Company details: http://www.itxperts.de/imprint.htm 
>> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: wip-auth

[Mark Nelson]

Hi Alex,

So far we don't really know, but suspect kernel/networking related 
things.  sysctl -a showed some differences in various vm/numa/ipv4 
settings.  Simply changing settings around might be a good first step, 
but if it's kernel related that might be tougher to track down.  The 
first step was to make sure it wasn't due to auth as that was suspect as 
well, but it seems to be an independent issue.

Mark

On 02/06/2015 01:45 AM, Alexandre DERUMIER wrote:
> Hi Mark,
>
> do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge)
>
>
> Not sure it's related, but I have found a bug report on ubuntu,
> about irqbalance not working correctly on ubuntu 14.04
> https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065
>
>
> ----- Mail original -----
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de>
> Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Jeudi 5 Février 2015 00:24:42
> Objet: Re: wip-auth
>
> Hi All,
>
> I completed some tests with wip-auth to the memstore on ubuntu earlier
> today. Basic gist of it is that the improvements in wip-auth help but
> don't quite get us to what can be achieved with auth disabled. RHEL7
> (without auth) continues to do very well in latency bound situations.
> Next up will be to see if how much this matters when testing against on
> SSDs.
>
> Here are the results:
>
> sync 4k object writes
> =====================
>
> Ceph OS Auth Avg Lat (ms) Avg IOPS
> ----------------------------------------------------------------
> master Ubuntu 14.04 Yes 0.99 1007
> Wip-auth Ubuntu 14.04 Yes 0.81 1237
> master Ubuntu 14.04 No 0.64 1549
> Wip-auth Ubuntu 14.04 No 0.65 1563
> master RHEL7 No 0.32 3158
>
> sync 4k object reads
> ====================
>
> Ceph OS Auth Avg Lat (ms) Avg IOPS
> ----------------------------------------------------------------
> master Ubuntu 14.04 Yes 0.59 1695
> Wip-auth Ubuntu 14.04 Yes 0.41 2409
> master Ubuntu 14.04 No 0.29 3425
> Wip-auth Ubuntu 14.04 No 0.29 3474
> master RHEL7 No 0.17 5853
>
> 256 concurrent 4k object writes
> ===============================
>
> Ceph OS Auth Avg Lat (ms) Avg IOPS
> ----------------------------------------------------------------
> master Ubuntu 14.04 Yes 40.39 6339
> Wip-auth Ubuntu 14.04 Yes 26.22 9763
> master Ubuntu 14.04 No 17.46 14662
> Wip-auth Ubuntu 14.04 No 17.34 14759
> master RHEL7 No 14.93 17139
>
> 256 concurrent 4k object reads
> ==============================
>
> Ceph OS Auth Avg Lat (ms) Avg IOPS
> ----------------------------------------------------------------
> master Ubuntu 14.04 Yes 31.47 8134
> Wip-auth Ubuntu 14.04 Yes 19.81 12922
> master Ubuntu 14.04 No 12.82 19968
> Wip-auth Ubuntu 14.04 No 12.75 20080
> master RHEL7 No 12.04 21257
>
> Mark
>
> On 01/30/2015 03:08 PM, Sage Weil wrote:
>> Hi Andreas,
>>
>> It looks like that was a stale sha1, but the newer one was also
>> broken. I've retested and it's working for me now. See latest wip-auth,
>> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.
>>
>> Thanks!
>> sage
>>
>> On Fri, 30 Jan 2015, Andreas Bluemle wrote:
>>
>>> Hi Sage,
>>>
>>> I tried to integrate wip-auth into my 0.91 build environment.
>>>
>>> I had not been able to start the cluster successfully: ceph-mon
>>> crashes with a segmentation fault in CryptoKey::encrypt()
>>> (see attachment).
>>>
>>> This happens when linking with libnss or libcryptopp (version 5.6.2).
>>>
>>> I created the patch to add wip-auth based on github
>>> pull request 3523 and was able to use this patch directly
>>> with 0.91 with only a minor adaptation for common/ceph_context.h:
>>> the 0.91 version of ceph_context.h did not know anything about
>>> the experimental "class CephContextObs".
>>>
>>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
>>>
>>> As my build environment is rpm, I had to modify the invocation
>>> of the configure script in the spec file instead of the do_autogen.sh
>>> script.
>>>
>>>
>>> Best Regards
>>>
>>> Andreas Bluemle
>>>
>>>
>>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST)
>>> Sage Weil <sweil@redhat.com> wrote:
>>>
>>>> I spent some time focusing on just CryptoKey::encrypt(). I
>>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
>>>>
>>>> cryptopp: 100000 encoded in 0.655651
>>>> libnss : 100000 encoded in 1.288786
>>>>
>>>> Ouch! With a (fixed) version of my earlier patch that avoids
>>>> uselessly copying the input buffer:
>>>>
>>>> 100000 encoded in 1.231977
>>>>
>>>> With a patch that puts the key structures in CryptoKey instead of
>>>> recreating them each time:
>>>>
>>>> 100000 encoded in 0.396208 -- ~70% improvement over original
>>>>
>>>> This is pushed to wip-auth. There's also a patch that caches key
>>>> structs for crypopp.. it now takes
>>>>
>>>> 100000 encoded in 0.440758 -- ~33% improvement over original
>>>>
>>>> (Not that almost anybody will ever care; we use libnss by default for
>>>> both rpm and deb distros.)
>>>>
>>>> So, yay, nss is now a bit faster. What I'm not completely certain
>>>> about is whether the structures I've preserved are truly stateless
>>>> (and can be shared across threads, etc.). They encrypt/decrypt
>>>> methods are const so, if the libraries are const-correct, it should
>>>> be fine... but perhaps someone familiar with nss and/or crypto++ can
>>>> review this?
>>>>
>>>> This is pushed to the latest wip-auth branch:
>>>>
>>>> https://github.com/ceph/ceph/commits/wip-auth
>>>>
>>>> Andreas and Stephen, what effect does this have on your numbers?
>>>>
>>>> Thanks!
>>>> sage
>>>>
>>>>
>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>>
>>>>> Good to know, I was wondering why the spec file defaulted to
>>>>> lib-nss.. the dpkg-build for debian packages just uses whatever
>>>>> configuration you had built, and I believe that will use
>>>>> libcryptopp if the dependency is installed on the build machine
>>>>> (last I looked).
>>>>>
>>>>> I forgot to mention the numbers below were based on v.91.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Stephen
>>>>>
>>>>> -----Original Message-----
>>>>> From: ceph-devel-owner@vger.kernel.org
>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
>>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
>>>>> Subject: RE: wip-auth
>>>>>
>>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>>>> I noticed that the spec file for building RPM's defaults to
>>>>>> building with libnss, instead of libcrypto++. Since the
>>>>>> measurements I'd done so far were from those RPM's I rebuilt with
>>>>>> libcrypto++.. so FWIW here is the difference between those two on
>>>>>> my system, memstore backend with a single OSD, and single
>>>>>> client.
>>>>>>
>>>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7
>>>>>> Kernel: 3.10.0-123.13.2.el7
>>>>>>
>>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench
>>>>>> libnss |
>>>>>> Cryptopp # QD IOPS Latency(ms) |
>>>>>> IOPS Latency(ms) IOPS Improvement % 16
>>>>>> 14432.57 1.11 | 18896.60 0.85
>>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados
>>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms)
>>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
>>>>>
>>>>> Yikes, 30%! I think this definitely worth some effort. We
>>>>> switched to libnss because it has the weird government
>>>>> certfiications that everyone wants and is more prevalent. crypto++
>>>>> is also not packaged for Red Hat distros at all (presumably for
>>>>> that reason).
>>>>>
>>>>> I suspect that most of the overhead is in the encryption context
>>>>> setup and can be avoided with a bit of effort..
>>>>>
>>>>> sage
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Stephen
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: ceph-devel-owner@vger.kernel.org
>>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>>>> Sent: Thursday, January 22, 2015 4:56 PM
>>>>>> To: andreas.bluemle@itxperts.de
>>>>>> Cc: ceph-devel@vger.kernel.org
>>>>>> Subject: wip-auth
>>>>>>
>>>>>> Hi Andreas,
>>>>>>
>>>>>> I took a look at the wip-auth I mentioned in the security call
>>>>>> last week... and the patch didn't work at all. Sorry if you
>>>>>> wasted any time trying it.
>>>>>>
>>>>>> Anyway, I fixed it up so that it actually worked and made one
>>>>>> other optimization. It would be great to hear what latencies you
>>>>>> measure with the changes in place.
>>>>>>
>>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if
>>>>>> you built cryptopp by default) to see if there is a difference.
>>>>>> There is a ton of boilerplate setting up encryption contexts and
>>>>>> key structures and so on that I suspect could be cached (perhaps
>>>>>> stashed in the CryptoKey struct?) with a bit of effort. See
>>>>>>
>>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
>>>>>>
>>>>>> sage
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>> -- To unsubscribe from this list: send the line "unsubscribe
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
>>> ITXperts GmbH http://www.itxperts.de
>>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
>>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>>>
>>> Company details: http://www.itxperts.de/imprint.htm
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: wip-auth

[Alexandre DERUMIER]

>>o far we don't really know, but suspect kernel/networking related 
>>things. sysctl -a showed some differences in various vm/numa/ipv4 
>>settings. Simply changing settings around might be a good first step, 
>>but if it's kernel related that might be tougher to track down. The 
>>first step was to make sure it wasn't due to auth as that was suspect as 
>>well, but it seems to be an independent issue. 

Ok thanks.
I'll try to compare too next month between debian and centos

----- Mail original -----
De: "Mark Nelson" <mnelson@redhat.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de>, "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 9 Février 2015 19:24:18
Objet: Re: wip-auth

Hi Alex, 

So far we don't really know, but suspect kernel/networking related 
things. sysctl -a showed some differences in various vm/numa/ipv4 
settings. Simply changing settings around might be a good first step, 
but if it's kernel related that might be tougher to track down. The 
first step was to make sure it wasn't due to auth as that was suspect as 
well, but it seems to be an independent issue. 

Mark 

On 02/06/2015 01:45 AM, Alexandre DERUMIER wrote: 
> Hi Mark, 
> 
> do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge) 
> 
> 
> Not sure it's related, but I have found a bug report on ubuntu, 
> about irqbalance not working correctly on ubuntu 14.04 
> https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065 
> 
> 
> ----- Mail original ----- 
> De: "Mark Nelson" <mark.nelson@inktank.com> 
> À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de> 
> Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org> 
> Envoyé: Jeudi 5 Février 2015 00:24:42 
> Objet: Re: wip-auth 
> 
> Hi All, 
> 
> I completed some tests with wip-auth to the memstore on ubuntu earlier 
> today. Basic gist of it is that the improvements in wip-auth help but 
> don't quite get us to what can be achieved with auth disabled. RHEL7 
> (without auth) continues to do very well in latency bound situations. 
> Next up will be to see if how much this matters when testing against on 
> SSDs. 
> 
> Here are the results: 
> 
> sync 4k object writes 
> ===================== 
> 
> Ceph OS Auth Avg Lat (ms) Avg IOPS 
> ---------------------------------------------------------------- 
> master Ubuntu 14.04 Yes 0.99 1007 
> Wip-auth Ubuntu 14.04 Yes 0.81 1237 
> master Ubuntu 14.04 No 0.64 1549 
> Wip-auth Ubuntu 14.04 No 0.65 1563 
> master RHEL7 No 0.32 3158 
> 
> sync 4k object reads 
> ==================== 
> 
> Ceph OS Auth Avg Lat (ms) Avg IOPS 
> ---------------------------------------------------------------- 
> master Ubuntu 14.04 Yes 0.59 1695 
> Wip-auth Ubuntu 14.04 Yes 0.41 2409 
> master Ubuntu 14.04 No 0.29 3425 
> Wip-auth Ubuntu 14.04 No 0.29 3474 
> master RHEL7 No 0.17 5853 
> 
> 256 concurrent 4k object writes 
> =============================== 
> 
> Ceph OS Auth Avg Lat (ms) Avg IOPS 
> ---------------------------------------------------------------- 
> master Ubuntu 14.04 Yes 40.39 6339 
> Wip-auth Ubuntu 14.04 Yes 26.22 9763 
> master Ubuntu 14.04 No 17.46 14662 
> Wip-auth Ubuntu 14.04 No 17.34 14759 
> master RHEL7 No 14.93 17139 
> 
> 256 concurrent 4k object reads 
> ============================== 
> 
> Ceph OS Auth Avg Lat (ms) Avg IOPS 
> ---------------------------------------------------------------- 
> master Ubuntu 14.04 Yes 31.47 8134 
> Wip-auth Ubuntu 14.04 Yes 19.81 12922 
> master Ubuntu 14.04 No 12.82 19968 
> Wip-auth Ubuntu 14.04 No 12.75 20080 
> master RHEL7 No 12.04 21257 
> 
> Mark 
> 
> On 01/30/2015 03:08 PM, Sage Weil wrote: 
>> Hi Andreas, 
>> 
>> It looks like that was a stale sha1, but the newer one was also 
>> broken. I've retested and it's working for me now. See latest wip-auth, 
>> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. 
>> 
>> Thanks! 
>> sage 
>> 
>> On Fri, 30 Jan 2015, Andreas Bluemle wrote: 
>> 
>>> Hi Sage, 
>>> 
>>> I tried to integrate wip-auth into my 0.91 build environment. 
>>> 
>>> I had not been able to start the cluster successfully: ceph-mon 
>>> crashes with a segmentation fault in CryptoKey::encrypt() 
>>> (see attachment). 
>>> 
>>> This happens when linking with libnss or libcryptopp (version 5.6.2). 
>>> 
>>> I created the patch to add wip-auth based on github 
>>> pull request 3523 and was able to use this patch directly 
>>> with 0.91 with only a minor adaptation for common/ceph_context.h: 
>>> the 0.91 version of ceph_context.h did not know anything about 
>>> the experimental "class CephContextObs". 
>>> 
>>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. 
>>> 
>>> As my build environment is rpm, I had to modify the invocation 
>>> of the configure script in the spec file instead of the do_autogen.sh 
>>> script. 
>>> 
>>> 
>>> Best Regards 
>>> 
>>> Andreas Bluemle 
>>> 
>>> 
>>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) 
>>> Sage Weil <sweil@redhat.com> wrote: 
>>> 
>>>> I spent some time focusing on just CryptoKey::encrypt(). I 
>>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, 
>>>> 
>>>> cryptopp: 100000 encoded in 0.655651 
>>>> libnss : 100000 encoded in 1.288786 
>>>> 
>>>> Ouch! With a (fixed) version of my earlier patch that avoids 
>>>> uselessly copying the input buffer: 
>>>> 
>>>> 100000 encoded in 1.231977 
>>>> 
>>>> With a patch that puts the key structures in CryptoKey instead of 
>>>> recreating them each time: 
>>>> 
>>>> 100000 encoded in 0.396208 -- ~70% improvement over original 
>>>> 
>>>> This is pushed to wip-auth. There's also a patch that caches key 
>>>> structs for crypopp.. it now takes 
>>>> 
>>>> 100000 encoded in 0.440758 -- ~33% improvement over original 
>>>> 
>>>> (Not that almost anybody will ever care; we use libnss by default for 
>>>> both rpm and deb distros.) 
>>>> 
>>>> So, yay, nss is now a bit faster. What I'm not completely certain 
>>>> about is whether the structures I've preserved are truly stateless 
>>>> (and can be shared across threads, etc.). They encrypt/decrypt 
>>>> methods are const so, if the libraries are const-correct, it should 
>>>> be fine... but perhaps someone familiar with nss and/or crypto++ can 
>>>> review this? 
>>>> 
>>>> This is pushed to the latest wip-auth branch: 
>>>> 
>>>> https://github.com/ceph/ceph/commits/wip-auth 
>>>> 
>>>> Andreas and Stephen, what effect does this have on your numbers? 
>>>> 
>>>> Thanks! 
>>>> sage 
>>>> 
>>>> 
>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: 
>>>> 
>>>>> Good to know, I was wondering why the spec file defaulted to 
>>>>> lib-nss.. the dpkg-build for debian packages just uses whatever 
>>>>> configuration you had built, and I believe that will use 
>>>>> libcryptopp if the dependency is installed on the build machine 
>>>>> (last I looked). 
>>>>> 
>>>>> I forgot to mention the numbers below were based on v.91. 
>>>>> 
>>>>> Thanks, 
>>>>> 
>>>>> Stephen 
>>>>> 
>>>>> -----Original Message----- 
>>>>> From: ceph-devel-owner@vger.kernel.org 
>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil 
>>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L 
>>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org 
>>>>> Subject: RE: wip-auth 
>>>>> 
>>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: 
>>>>>> I noticed that the spec file for building RPM's defaults to 
>>>>>> building with libnss, instead of libcrypto++. Since the 
>>>>>> measurements I'd done so far were from those RPM's I rebuilt with 
>>>>>> libcrypto++.. so FWIW here is the difference between those two on 
>>>>>> my system, memstore backend with a single OSD, and single 
>>>>>> client. 
>>>>>> 
>>>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 
>>>>>> Kernel: 3.10.0-123.13.2.el7 
>>>>>> 
>>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench 
>>>>>> libnss | 
>>>>>> Cryptopp # QD IOPS Latency(ms) | 
>>>>>> IOPS Latency(ms) IOPS Improvement % 16 
>>>>>> 14432.57 1.11 | 18896.60 0.85 
>>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados 
>>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) 
>>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% 
>>>>> 
>>>>> Yikes, 30%! I think this definitely worth some effort. We 
>>>>> switched to libnss because it has the weird government 
>>>>> certfiications that everyone wants and is more prevalent. crypto++ 
>>>>> is also not packaged for Red Hat distros at all (presumably for 
>>>>> that reason). 
>>>>> 
>>>>> I suspect that most of the overhead is in the encryption context 
>>>>> setup and can be avoided with a bit of effort.. 
>>>>> 
>>>>> sage 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks, 
>>>>>> 
>>>>>> Stephen 
>>>>>> 
>>>>>> -----Original Message----- 
>>>>>> From: ceph-devel-owner@vger.kernel.org 
>>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil 
>>>>>> Sent: Thursday, January 22, 2015 4:56 PM 
>>>>>> To: andreas.bluemle@itxperts.de 
>>>>>> Cc: ceph-devel@vger.kernel.org 
>>>>>> Subject: wip-auth 
>>>>>> 
>>>>>> Hi Andreas, 
>>>>>> 
>>>>>> I took a look at the wip-auth I mentioned in the security call 
>>>>>> last week... and the patch didn't work at all. Sorry if you 
>>>>>> wasted any time trying it. 
>>>>>> 
>>>>>> Anyway, I fixed it up so that it actually worked and made one 
>>>>>> other optimization. It would be great to hear what latencies you 
>>>>>> measure with the changes in place. 
>>>>>> 
>>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if 
>>>>>> you built cryptopp by default) to see if there is a difference. 
>>>>>> There is a ton of boilerplate setting up encryption contexts and 
>>>>>> key structures and so on that I suspect could be cached (perhaps 
>>>>>> stashed in the CryptoKey struct?) with a bit of effort. See 
>>>>>> 
>>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 
>>>>>> 
>>>>>> sage 
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>>>> 
>>>>>> 
>>>>> -- 
>>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>>> -- To unsubscribe from this list: send the line "unsubscribe 
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de 
>>> ITXperts GmbH http://www.itxperts.de 
>>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 
>>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 
>>> 
>>> Company details: http://www.itxperts.de/imprint.htm 
>>> 
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>> the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: wip-auth

[Blinick, Stephen L]

Hi -- was running some baselines on 0.93 and wanted to check, will wip-auth changes go into Hamer or in a later release?   (Sorry if I missed the discussion in one of the perf meetings last month)

Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Wednesday, February 4, 2015 4:25 PM
To: Sage Weil; Andreas Bluemle
Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org
Subject: Re: wip-auth

Hi All,

I completed some tests with wip-auth to the memstore on ubuntu earlier today.  Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled.  RHEL7 (without auth) continues to do very well in latency bound situations. 
Next up will be to see if how much this matters when testing against on SSDs.

Here are the results:

sync 4k object writes
=====================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	0.99		1007
Wip-auth	Ubuntu 14.04	Yes	0.81		1237
master		Ubuntu 14.04	No	0.64		1549
Wip-auth	Ubuntu 14.04	No	0.65		1563
master		RHEL7		No	0.32		3158

sync 4k object reads
====================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	0.59		1695
Wip-auth	Ubuntu 14.04	Yes	0.41		2409
master		Ubuntu 14.04	No	0.29		3425
Wip-auth	Ubuntu 14.04	No	0.29		3474
master		RHEL7		No	0.17		5853

256 concurrent 4k object writes
===============================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	40.39		6339
Wip-auth	Ubuntu 14.04	Yes	26.22		9763
master		Ubuntu 14.04	No	17.46		14662
Wip-auth	Ubuntu 14.04	No	17.34		14759
master		RHEL7		No	14.93		17139

256 concurrent 4k object reads
==============================

Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
----------------------------------------------------------------
master		Ubuntu 14.04	Yes	31.47		8134
Wip-auth	Ubuntu 14.04	Yes	19.81		12922
master		Ubuntu 14.04	No	12.82		19968
Wip-auth	Ubuntu 14.04	No	12.75		20080
master		RHEL7		No	12.04		21257

Mark

On 01/30/2015 03:08 PM, Sage Weil wrote:
> Hi Andreas,
>
> It looks like that was a stale sha1, but the newer one was also 
> broken.  I've retested and it's working for me now.  See latest 
> wip-auth,
> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.
>
> Thanks!
> sage
>
> On Fri, 30 Jan 2015, Andreas Bluemle wrote:
>
>> Hi Sage,
>>
>> I tried to integrate wip-auth into my 0.91 build environment.
>>
>> I had not been able to start the cluster successfully: ceph-mon 
>> crashes with a segmentation fault in CryptoKey::encrypt() (see 
>> attachment).
>>
>> This happens when linking with libnss or libcryptopp (version 5.6.2).
>>
>> I created the patch to add wip-auth based on github pull request 3523 
>> and was able to use this patch directly with 0.91 with only a minor 
>> adaptation for common/ceph_context.h:
>> the 0.91 version of ceph_context.h did not know anything about the 
>> experimental "class CephContextObs".
>>
>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
>>
>> As my build environment is rpm, I had to modify the invocation of the 
>> configure script in the spec file instead of the do_autogen.sh 
>> script.
>>
>>
>> Best Regards
>>
>> Andreas Bluemle
>>
>>
>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@redhat.com> 
>> wrote:
>>
>>> I spent some time focusing on just CryptoKey::encrypt().  I 
>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
>>>
>>>   cryptopp: 100000 encoded in 0.655651
>>>   libnss  : 100000 encoded in 1.288786
>>>
>>> Ouch!  With a (fixed) version of my earlier patch that avoids 
>>> uselessly copying the input buffer:
>>>
>>>   100000 encoded in 1.231977
>>>
>>> With a patch that puts the key structures in CryptoKey instead of 
>>> recreating them each time:
>>>
>>>   100000 encoded in 0.396208  -- ~70% improvement over original
>>>
>>> This is pushed to wip-auth.  There's also a patch that caches key 
>>> structs for crypopp.. it now takes
>>>
>>>   100000 encoded in 0.440758  -- ~33% improvement over original
>>>
>>> (Not that almost anybody will ever care; we use libnss by default 
>>> for both rpm and deb distros.)
>>>
>>> So, yay, nss is now a bit faster.  What I'm not completely certain 
>>> about is whether the structures I've preserved are truly stateless 
>>> (and can be shared across threads, etc.).  They encrypt/decrypt 
>>> methods are const so, if the libraries are const-correct, it should 
>>> be fine... but perhaps someone familiar with nss and/or crypto++ can 
>>> review this?
>>>
>>> This is pushed to the latest wip-auth branch:
>>>
>>> 	https://github.com/ceph/ceph/commits/wip-auth
>>>
>>> Andreas and Stephen, what effect does this have on your numbers?
>>>
>>> Thanks!
>>> sage
>>>
>>>
>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>
>>>> Good to know, I was wondering why the spec file defaulted to 
>>>> lib-nss.. the dpkg-build for debian packages just uses whatever 
>>>> configuration you had built, and I believe that will use 
>>>> libcryptopp if the dependency is installed on the build machine 
>>>> (last I looked).
>>>>
>>>> I forgot to mention the numbers below were based on v.91.
>>>>
>>>> Thanks,
>>>>
>>>> Stephen
>>>>
>>>> -----Original Message-----
>>>> From: ceph-devel-owner@vger.kernel.org 
>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
>>>> Subject: RE: wip-auth
>>>>
>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
>>>>> I noticed that the spec file for building RPM's defaults to 
>>>>> building with libnss, instead of libcrypto++.  Since the 
>>>>> measurements I'd done so far were from those RPM's I rebuilt with
>>>>> libcrypto++.. so FWIW here is the difference between those two on
>>>>> my system, memstore backend with a single OSD, and single client.
>>>>>
>>>>> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
>>>>> Kernel: 3.10.0-123.13.2.el7
>>>>>
>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench
>>>>> 	libnss		            |
>>>>> Cryptopp # QD	IOPS	Latency(ms)   |
>>>>> IOPS	Latency(ms)	IOPS Improvement % 16
>>>>> 14432.57	1.11    |	18896.60	0.85
>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD 
>>>>> IOPS Latency(ms)  | IOPS Latency(ms) IOPS Improvement % 16 
>>>>> 19532.53 0.82 | 25708.70 0.62 31.62%
>>>>
>>>> Yikes, 30%!  I think this definitely worth some effort.  We 
>>>> switched to libnss because it has the weird government 
>>>> certfiications that everyone wants and is more prevalent.  crypto++ 
>>>> is also not packaged for Red Hat distros at all (presumably for 
>>>> that reason).
>>>>
>>>> I suspect that most of the overhead is in the encryption context 
>>>> setup and can be avoided with a bit of effort..
>>>>
>>>> sage
>>>>
>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Stephen
>>>>>
>>>>> -----Original Message-----
>>>>> From: ceph-devel-owner@vger.kernel.org 
>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>>>> Sent: Thursday, January 22, 2015 4:56 PM
>>>>> To: andreas.bluemle@itxperts.de
>>>>> Cc: ceph-devel@vger.kernel.org
>>>>> Subject: wip-auth
>>>>>
>>>>> Hi Andreas,
>>>>>
>>>>> I took a look at the wip-auth I mentioned in the security call 
>>>>> last week... and the patch didn't work at all.  Sorry if you 
>>>>> wasted any time trying it.
>>>>>
>>>>> Anyway, I fixed it up so that it actually worked and made one 
>>>>> other optimization.  It would be great to hear what latencies you 
>>>>> measure with the changes in place.
>>>>>
>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if 
>>>>> you built cryptopp by default) to see if there is a difference.
>>>>> There is a ton of boilerplate setting up encryption contexts and 
>>>>> key structures and so on that I suspect could be cached (perhaps 
>>>>> stashed in the CryptoKey struct?) with a bit of effort.  See
>>>>>
>>>>> 	
>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L2
>>>>> 13
>>>>>
>>>>> sage
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> -- To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
>> ITXperts GmbH                       http://www.itxperts.de
>> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
>> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>>
>> Company details: http://www.itxperts.de/imprint.htm
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: wip-auth

[Sage Weil]

On Tue, 10 Mar 2015, Blinick, Stephen L wrote:
> Hi -- was running some baselines on 0.93 and wanted to check, will 
> wip-auth changes go into Hamer or in a later release?  (Sorry if I 
> missed the discussion in one of the perf meetings last month)

It will go in after hammer.  We can backport as needed.

sage

> 
> Thanks,
> 
> Stephen
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Wednesday, February 4, 2015 4:25 PM
> To: Sage Weil; Andreas Bluemle
> Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org
> Subject: Re: wip-auth
> 
> Hi All,
> 
> I completed some tests with wip-auth to the memstore on ubuntu earlier today.  Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled.  RHEL7 (without auth) continues to do very well in latency bound situations. 
> Next up will be to see if how much this matters when testing against on SSDs.
> 
> Here are the results:
> 
> sync 4k object writes
> =====================
> 
> Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
> ----------------------------------------------------------------
> master		Ubuntu 14.04	Yes	0.99		1007
> Wip-auth	Ubuntu 14.04	Yes	0.81		1237
> master		Ubuntu 14.04	No	0.64		1549
> Wip-auth	Ubuntu 14.04	No	0.65		1563
> master		RHEL7		No	0.32		3158
> 
> sync 4k object reads
> ====================
> 
> Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
> ----------------------------------------------------------------
> master		Ubuntu 14.04	Yes	0.59		1695
> Wip-auth	Ubuntu 14.04	Yes	0.41		2409
> master		Ubuntu 14.04	No	0.29		3425
> Wip-auth	Ubuntu 14.04	No	0.29		3474
> master		RHEL7		No	0.17		5853
> 
> 256 concurrent 4k object writes
> ===============================
> 
> Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
> ----------------------------------------------------------------
> master		Ubuntu 14.04	Yes	40.39		6339
> Wip-auth	Ubuntu 14.04	Yes	26.22		9763
> master		Ubuntu 14.04	No	17.46		14662
> Wip-auth	Ubuntu 14.04	No	17.34		14759
> master		RHEL7		No	14.93		17139
> 
> 256 concurrent 4k object reads
> ==============================
> 
> Ceph		OS		Auth	Avg Lat (ms)	Avg IOPS
> ----------------------------------------------------------------
> master		Ubuntu 14.04	Yes	31.47		8134
> Wip-auth	Ubuntu 14.04	Yes	19.81		12922
> master		Ubuntu 14.04	No	12.82		19968
> Wip-auth	Ubuntu 14.04	No	12.75		20080
> master		RHEL7		No	12.04		21257
> 
> Mark
> 
> On 01/30/2015 03:08 PM, Sage Weil wrote:
> > Hi Andreas,
> >
> > It looks like that was a stale sha1, but the newer one was also 
> > broken.  I've retested and it's working for me now.  See latest 
> > wip-auth,
> > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.
> >
> > Thanks!
> > sage
> >
> > On Fri, 30 Jan 2015, Andreas Bluemle wrote:
> >
> >> Hi Sage,
> >>
> >> I tried to integrate wip-auth into my 0.91 build environment.
> >>
> >> I had not been able to start the cluster successfully: ceph-mon 
> >> crashes with a segmentation fault in CryptoKey::encrypt() (see 
> >> attachment).
> >>
> >> This happens when linking with libnss or libcryptopp (version 5.6.2).
> >>
> >> I created the patch to add wip-auth based on github pull request 3523 
> >> and was able to use this patch directly with 0.91 with only a minor 
> >> adaptation for common/ceph_context.h:
> >> the 0.91 version of ceph_context.h did not know anything about the 
> >> experimental "class CephContextObs".
> >>
> >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
> >>
> >> As my build environment is rpm, I had to modify the invocation of the 
> >> configure script in the spec file instead of the do_autogen.sh 
> >> script.
> >>
> >>
> >> Best Regards
> >>
> >> Andreas Bluemle
> >>
> >>
> >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@redhat.com> 
> >> wrote:
> >>
> >>> I spent some time focusing on just CryptoKey::encrypt().  I 
> >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
> >>>
> >>>   cryptopp: 100000 encoded in 0.655651
> >>>   libnss  : 100000 encoded in 1.288786
> >>>
> >>> Ouch!  With a (fixed) version of my earlier patch that avoids 
> >>> uselessly copying the input buffer:
> >>>
> >>>   100000 encoded in 1.231977
> >>>
> >>> With a patch that puts the key structures in CryptoKey instead of 
> >>> recreating them each time:
> >>>
> >>>   100000 encoded in 0.396208  -- ~70% improvement over original
> >>>
> >>> This is pushed to wip-auth.  There's also a patch that caches key 
> >>> structs for crypopp.. it now takes
> >>>
> >>>   100000 encoded in 0.440758  -- ~33% improvement over original
> >>>
> >>> (Not that almost anybody will ever care; we use libnss by default 
> >>> for both rpm and deb distros.)
> >>>
> >>> So, yay, nss is now a bit faster.  What I'm not completely certain 
> >>> about is whether the structures I've preserved are truly stateless 
> >>> (and can be shared across threads, etc.).  They encrypt/decrypt 
> >>> methods are const so, if the libraries are const-correct, it should 
> >>> be fine... but perhaps someone familiar with nss and/or crypto++ can 
> >>> review this?
> >>>
> >>> This is pushed to the latest wip-auth branch:
> >>>
> >>> 	https://github.com/ceph/ceph/commits/wip-auth
> >>>
> >>> Andreas and Stephen, what effect does this have on your numbers?
> >>>
> >>> Thanks!
> >>> sage
> >>>
> >>>
> >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> >>>
> >>>> Good to know, I was wondering why the spec file defaulted to 
> >>>> lib-nss.. the dpkg-build for debian packages just uses whatever 
> >>>> configuration you had built, and I believe that will use 
> >>>> libcryptopp if the dependency is installed on the build machine 
> >>>> (last I looked).
> >>>>
> >>>> I forgot to mention the numbers below were based on v.91.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Stephen
> >>>>
> >>>> -----Original Message-----
> >>>> From: ceph-devel-owner@vger.kernel.org 
> >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
> >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org
> >>>> Subject: RE: wip-auth
> >>>>
> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
> >>>>> I noticed that the spec file for building RPM's defaults to 
> >>>>> building with libnss, instead of libcrypto++.  Since the 
> >>>>> measurements I'd done so far were from those RPM's I rebuilt with
> >>>>> libcrypto++.. so FWIW here is the difference between those two on
> >>>>> my system, memstore backend with a single OSD, and single client.
> >>>>>
> >>>>> Dual socket Xeon E5 2620v3, 64GB Memory,  RHEL7
> >>>>> Kernel: 3.10.0-123.13.2.el7
> >>>>>
> >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench
> >>>>> 	libnss		            |
> >>>>> Cryptopp # QD	IOPS	Latency(ms)   |
> >>>>> IOPS	Latency(ms)	IOPS Improvement % 16
> >>>>> 14432.57	1.11    |	18896.60	0.85
> >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD 
> >>>>> IOPS Latency(ms)  | IOPS Latency(ms) IOPS Improvement % 16 
> >>>>> 19532.53 0.82 | 25708.70 0.62 31.62%
> >>>>
> >>>> Yikes, 30%!  I think this definitely worth some effort.  We 
> >>>> switched to libnss because it has the weird government 
> >>>> certfiications that everyone wants and is more prevalent.  crypto++ 
> >>>> is also not packaged for Red Hat distros at all (presumably for 
> >>>> that reason).
> >>>>
> >>>> I suspect that most of the overhead is in the encryption context 
> >>>> setup and can be avoided with a bit of effort..
> >>>>
> >>>> sage
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Stephen
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: ceph-devel-owner@vger.kernel.org 
> >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> >>>>> Sent: Thursday, January 22, 2015 4:56 PM
> >>>>> To: andreas.bluemle@itxperts.de
> >>>>> Cc: ceph-devel@vger.kernel.org
> >>>>> Subject: wip-auth
> >>>>>
> >>>>> Hi Andreas,
> >>>>>
> >>>>> I took a look at the wip-auth I mentioned in the security call 
> >>>>> last week... and the patch didn't work at all.  Sorry if you 
> >>>>> wasted any time trying it.
> >>>>>
> >>>>> Anyway, I fixed it up so that it actually worked and made one 
> >>>>> other optimization.  It would be great to hear what latencies you 
> >>>>> measure with the changes in place.
> >>>>>
> >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if 
> >>>>> you built cryptopp by default) to see if there is a difference.
> >>>>> There is a ton of boilerplate setting up encryption contexts and 
> >>>>> key structures and so on that I suspect could be cached (perhaps 
> >>>>> stashed in the CryptoKey struct?) with a bit of effort.  See
> >>>>>
> >>>>> 	
> >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L2
> >>>>> 13
> >>>>>
> >>>>> sage
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe 
> >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe 
> >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>> -- To unsubscribe from this list: send the line "unsubscribe 
> >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> >> ITXperts GmbH                       http://www.itxperts.de
> >> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> >> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> >>
> >> Company details: http://www.itxperts.de/imprint.htm
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>