* wip-auth @ 2015-01-22 23:55 Sage Weil 2015-01-26 6:20 ` wip-auth Blinick, Stephen L 0 siblings, 1 reply; 15+ messages in thread From: Sage Weil @ 2015-01-22 23:55 UTC (permalink / raw) To: andreas.bluemle; +Cc: ceph-devel Hi Andreas, I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. Also, it might be worth trying --with-cryptopp (or --with-nss if you built cryptopp by default) to see if there is a difference. There is a ton of boilerplate setting up encryption contexts and key structures and so on that I suspect could be cached (perhaps stashed in the CryptoKey struct?) with a bit of effort. See https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 sage ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-01-22 23:55 wip-auth Sage Weil @ 2015-01-26 6:20 ` Blinick, Stephen L 2015-01-26 17:23 ` wip-auth Sage Weil 0 siblings, 1 reply; 15+ messages in thread From: Blinick, Stephen L @ 2015-01-26 6:20 UTC (permalink / raw) To: Sage Weil, andreas.bluemle; +Cc: ceph-devel I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 Kernel: 3.10.0-123.13.2.el7 100% 4K Writes, 1xOSD w/ Rados Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % 16 14432.57 1.11 | 18896.60 0.85 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% Thanks, Stephen -----Original Message----- From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil Sent: Thursday, January 22, 2015 4:56 PM To: andreas.bluemle@itxperts.de Cc: ceph-devel@vger.kernel.org Subject: wip-auth Hi Andreas, I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. Also, it might be worth trying --with-cryptopp (or --with-nss if you built cryptopp by default) to see if there is a difference. There is a ton of boilerplate setting up encryption contexts and key structures and so on that I suspect could be cached (perhaps stashed in the CryptoKey struct?) with a bit of effort. See https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-01-26 6:20 ` wip-auth Blinick, Stephen L @ 2015-01-26 17:23 ` Sage Weil 2015-01-26 18:00 ` wip-auth Blinick, Stephen L 0 siblings, 1 reply; 15+ messages in thread From: Sage Weil @ 2015-01-26 17:23 UTC (permalink / raw) To: Blinick, Stephen L; +Cc: andreas.bluemle, ceph-devel On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. > > Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > Kernel: 3.10.0-123.13.2.el7 > > 100% 4K Writes, 1xOSD w/ Rados Bench > libnss | Cryptopp > # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % > 16 14432.57 1.11 | 18896.60 0.85 30.93% > > 100% 4K Reads, 1xOSD w/ Rados Bench > libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS > Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% Yikes, 30%! I think this definitely worth some effort. We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent. crypto++ is also not packaged for Red Hat distros at all (presumably for that reason). I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort.. sage > > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Thursday, January 22, 2015 4:56 PM > To: andreas.bluemle@itxperts.de > Cc: ceph-devel@vger.kernel.org > Subject: wip-auth > > Hi Andreas, > > I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. > > Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. > > Also, it might be worth trying --with-cryptopp (or --with-nss if you built cryptopp by default) to see if there is a difference. There is a ton of boilerplate setting up encryption contexts and key structures and so on that I suspect could be cached (perhaps stashed in the CryptoKey struct?) with a bit of effort. See > > https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 > > sage > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-01-26 17:23 ` wip-auth Sage Weil @ 2015-01-26 18:00 ` Blinick, Stephen L 2015-01-26 18:16 ` wip-auth Mark Nelson 2015-01-27 17:18 ` wip-auth Sage Weil 0 siblings, 2 replies; 15+ messages in thread From: Blinick, Stephen L @ 2015-01-26 18:00 UTC (permalink / raw) To: Sage Weil; +Cc: andreas.bluemle, ceph-devel Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked). I forgot to mention the numbers below were based on v.91. Thanks, Stephen -----Original Message----- From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org Subject: RE: wip-auth On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. > > Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > Kernel: 3.10.0-123.13.2.el7 > > 100% 4K Writes, 1xOSD w/ Rados Bench > libnss | Cryptopp > # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % > 16 14432.57 1.11 | 18896.60 0.85 30.93% > > 100% 4K Reads, 1xOSD w/ Rados Bench > libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS > Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% Yikes, 30%! I think this definitely worth some effort. We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent. crypto++ is also not packaged for Red Hat distros at all (presumably for that reason). I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort.. sage > > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Thursday, January 22, 2015 4:56 PM > To: andreas.bluemle@itxperts.de > Cc: ceph-devel@vger.kernel.org > Subject: wip-auth > > Hi Andreas, > > I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. > > Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. > > Also, it might be worth trying --with-cryptopp (or --with-nss if you > built cryptopp by default) to see if there is a difference. There is > a ton of boilerplate setting up encryption contexts and key structures > and so on that I suspect could be cached (perhaps stashed in the > CryptoKey struct?) with a bit of effort. See > > https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 > > sage > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: wip-auth 2015-01-26 18:00 ` wip-auth Blinick, Stephen L @ 2015-01-26 18:16 ` Mark Nelson 2015-01-28 1:10 ` wip-auth Blinick, Stephen L 2015-01-27 17:18 ` wip-auth Sage Weil 1 sibling, 1 reply; 15+ messages in thread From: Mark Nelson @ 2015-01-26 18:16 UTC (permalink / raw) To: Blinick, Stephen L, Sage Weil; +Cc: andreas.bluemle, ceph-devel Hi Stephen, Does this explain the results you were seeing earlier with the memstore testing? Mark On 01/26/2015 12:00 PM, Blinick, Stephen L wrote: > Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked). > > I forgot to mention the numbers below were based on v.91. > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Monday, January 26, 2015 10:24 AM > To: Blinick, Stephen L > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > Subject: RE: wip-auth > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. >> >> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >> Kernel: 3.10.0-123.13.2.el7 >> >> 100% 4K Writes, 1xOSD w/ Rados Bench >> libnss | Cryptopp >> # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % >> 16 14432.57 1.11 | 18896.60 0.85 30.93% >> >> 100% 4K Reads, 1xOSD w/ Rados Bench >> libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS >> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% > > Yikes, 30%! I think this definitely worth some effort. We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent. crypto++ is also not packaged for Red Hat distros at all (presumably for that reason). > > I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort.. > > sage > > >> >> >> Thanks, >> >> Stephen >> >> -----Original Message----- >> From: ceph-devel-owner@vger.kernel.org >> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >> Sent: Thursday, January 22, 2015 4:56 PM >> To: andreas.bluemle@itxperts.de >> Cc: ceph-devel@vger.kernel.org >> Subject: wip-auth >> >> Hi Andreas, >> >> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. >> >> Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. >> >> Also, it might be worth trying --with-cryptopp (or --with-nss if you >> built cryptopp by default) to see if there is a difference. There is >> a ton of boilerplate setting up encryption contexts and key structures >> and so on that I suspect could be cached (perhaps stashed in the >> CryptoKey struct?) with a bit of effort. See >> >> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >> >> sage >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majordomo@vger.kernel.org More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-01-26 18:16 ` wip-auth Mark Nelson @ 2015-01-28 1:10 ` Blinick, Stephen L 0 siblings, 0 replies; 15+ messages in thread From: Blinick, Stephen L @ 2015-01-28 1:10 UTC (permalink / raw) To: mnelson, Sage Weil; +Cc: andreas.bluemle, ceph-devel Hi Mark -- it doesn't, but it helps explain why there's some variance in all the various measurements. I've only been running with the various debug settings off, but message signing, crc, authentication, etc at defaults. I know some of the other results are with everything off, and that seems to have a large impact. Somewhere when switching versions, libnss became default, and so I was comparing RHEL7 w/ libnss to Ubuntu w/ Cryptopp. When I switched to Cryptopp with RHEL7 below the numbers in my environment improved again. However, since libnss is the direction, I'll stick to that (and hopefully make back those improvements and more in the latest wip-auth commits). Seems complex enough it's worth talking through verbally :) Thanks, Stephen -----Original Message----- From: Mark Nelson [mailto:mark.nelson@inktank.com] Sent: Monday, January 26, 2015 11:16 AM To: Blinick, Stephen L; Sage Weil Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org Subject: Re: wip-auth Hi Stephen, Does this explain the results you were seeing earlier with the memstore testing? Mark On 01/26/2015 12:00 PM, Blinick, Stephen L wrote: > Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked). > > I forgot to mention the numbers below were based on v.91. > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Monday, January 26, 2015 10:24 AM > To: Blinick, Stephen L > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > Subject: RE: wip-auth > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >> I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. >> >> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >> Kernel: 3.10.0-123.13.2.el7 >> >> 100% 4K Writes, 1xOSD w/ Rados Bench >> libnss | Cryptopp >> # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % >> 16 14432.57 1.11 | 18896.60 0.85 30.93% >> >> 100% 4K Reads, 1xOSD w/ Rados Bench >> libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS >> Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% > > Yikes, 30%! I think this definitely worth some effort. We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent. crypto++ is also not packaged for Red Hat distros at all (presumably for that reason). > > I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort.. > > sage > > >> >> >> Thanks, >> >> Stephen >> >> -----Original Message----- >> From: ceph-devel-owner@vger.kernel.org >> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >> Sent: Thursday, January 22, 2015 4:56 PM >> To: andreas.bluemle@itxperts.de >> Cc: ceph-devel@vger.kernel.org >> Subject: wip-auth >> >> Hi Andreas, >> >> I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. >> >> Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. >> >> Also, it might be worth trying --with-cryptopp (or --with-nss if you >> built cryptopp by default) to see if there is a difference. There is >> a ton of boilerplate setting up encryption contexts and key >> structures and so on that I suspect could be cached (perhaps stashed >> in the CryptoKey struct?) with a bit of effort. See >> >> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >> >> sage >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majordomo@vger.kernel.org More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-01-26 18:00 ` wip-auth Blinick, Stephen L 2015-01-26 18:16 ` wip-auth Mark Nelson @ 2015-01-27 17:18 ` Sage Weil 2015-01-30 7:10 ` wip-auth Andreas Bluemle 1 sibling, 1 reply; 15+ messages in thread From: Sage Weil @ 2015-01-27 17:18 UTC (permalink / raw) To: Blinick, Stephen L; +Cc: andreas.bluemle, ceph-devel I spent some time focusing on just CryptoKey::encrypt(). I benchmarked 100,000 encrypts of 128 bytes and got, at baseline, cryptopp: 100000 encoded in 0.655651 libnss : 100000 encoded in 1.288786 Ouch! With a (fixed) version of my earlier patch that avoids uselessly copying the input buffer: 100000 encoded in 1.231977 With a patch that puts the key structures in CryptoKey instead of recreating them each time: 100000 encoded in 0.396208 -- ~70% improvement over original This is pushed to wip-auth. There's also a patch that caches key structs for crypopp.. it now takes 100000 encoded in 0.440758 -- ~33% improvement over original (Not that almost anybody will ever care; we use libnss by default for both rpm and deb distros.) So, yay, nss is now a bit faster. What I'm not completely certain about is whether the structures I've preserved are truly stateless (and can be shared across threads, etc.). They encrypt/decrypt methods are const so, if the libraries are const-correct, it should be fine... but perhaps someone familiar with nss and/or crypto++ can review this? This is pushed to the latest wip-auth branch: https://github.com/ceph/ceph/commits/wip-auth Andreas and Stephen, what effect does this have on your numbers? Thanks! sage On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > Good to know, I was wondering why the spec file defaulted to lib-nss.. the dpkg-build for debian packages just uses whatever configuration you had built, and I believe that will use libcryptopp if the dependency is installed on the build machine (last I looked). > > I forgot to mention the numbers below were based on v.91. > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Monday, January 26, 2015 10:24 AM > To: Blinick, Stephen L > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > Subject: RE: wip-auth > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > > I noticed that the spec file for building RPM's defaults to building with libnss, instead of libcrypto++. Since the measurements I'd done so far were from those RPM's I rebuilt with libcrypto++.. so FWIW here is the difference between those two on my system, memstore backend with a single OSD, and single client. > > > > Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > > Kernel: 3.10.0-123.13.2.el7 > > > > 100% 4K Writes, 1xOSD w/ Rados Bench > > libnss | Cryptopp > > # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % > > 16 14432.57 1.11 | 18896.60 0.85 30.93% > > > > 100% 4K Reads, 1xOSD w/ Rados Bench > > libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) IOPS > > Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% > > Yikes, 30%! I think this definitely worth some effort. We switched to libnss because it has the weird government certfiications that everyone wants and is more prevalent. crypto++ is also not packaged for Red Hat distros at all (presumably for that reason). > > I suspect that most of the overhead is in the encryption context setup and can be avoided with a bit of effort.. > > sage > > > > > > > > Thanks, > > > > Stephen > > > > -----Original Message----- > > From: ceph-devel-owner@vger.kernel.org > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > > Sent: Thursday, January 22, 2015 4:56 PM > > To: andreas.bluemle@itxperts.de > > Cc: ceph-devel@vger.kernel.org > > Subject: wip-auth > > > > Hi Andreas, > > > > I took a look at the wip-auth I mentioned in the security call last week... and the patch didn't work at all. Sorry if you wasted any time trying it. > > > > Anyway, I fixed it up so that it actually worked and made one other optimization. It would be great to hear what latencies you measure with the changes in place. > > > > Also, it might be worth trying --with-cryptopp (or --with-nss if you > > built cryptopp by default) to see if there is a difference. There is > > a ton of boilerplate setting up encryption contexts and key structures > > and so on that I suspect could be cached (perhaps stashed in the > > CryptoKey struct?) with a bit of effort. See > > > > https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 > > > > sage > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@vger.kernel.org More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: wip-auth 2015-01-27 17:18 ` wip-auth Sage Weil @ 2015-01-30 7:10 ` Andreas Bluemle 2015-01-30 21:08 ` wip-auth Sage Weil 0 siblings, 1 reply; 15+ messages in thread From: Andreas Bluemle @ 2015-01-30 7:10 UTC (permalink / raw) To: Sage Weil; +Cc: Blinick, Stephen L, ceph-devel [-- Attachment #1: Type: text/plain, Size: 6487 bytes --] Hi Sage, I tried to integrate wip-auth into my 0.91 build environment. I had not been able to start the cluster successfully: ceph-mon crashes with a segmentation fault in CryptoKey::encrypt() (see attachment). This happens when linking with libnss or libcryptopp (version 5.6.2). I created the patch to add wip-auth based on github pull request 3523 and was able to use this patch directly with 0.91 with only a minor adaptation for common/ceph_context.h: the 0.91 version of ceph_context.h did not know anything about the experimental "class CephContextObs". wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. As my build environment is rpm, I had to modify the invocation of the configure script in the spec file instead of the do_autogen.sh script. Best Regards Andreas Bluemle On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@redhat.com> wrote: > I spent some time focusing on just CryptoKey::encrypt(). I > benchmarked 100,000 encrypts of 128 bytes and got, at baseline, > > cryptopp: 100000 encoded in 0.655651 > libnss : 100000 encoded in 1.288786 > > Ouch! With a (fixed) version of my earlier patch that avoids > uselessly copying the input buffer: > > 100000 encoded in 1.231977 > > With a patch that puts the key structures in CryptoKey instead of > recreating them each time: > > 100000 encoded in 0.396208 -- ~70% improvement over original > > This is pushed to wip-auth. There's also a patch that caches key > structs for crypopp.. it now takes > > 100000 encoded in 0.440758 -- ~33% improvement over original > > (Not that almost anybody will ever care; we use libnss by default for > both rpm and deb distros.) > > So, yay, nss is now a bit faster. What I'm not completely certain > about is whether the structures I've preserved are truly stateless > (and can be shared across threads, etc.). They encrypt/decrypt > methods are const so, if the libraries are const-correct, it should > be fine... but perhaps someone familiar with nss and/or crypto++ can > review this? > > This is pushed to the latest wip-auth branch: > > https://github.com/ceph/ceph/commits/wip-auth > > Andreas and Stephen, what effect does this have on your numbers? > > Thanks! > sage > > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > > > Good to know, I was wondering why the spec file defaulted to > > lib-nss.. the dpkg-build for debian packages just uses whatever > > configuration you had built, and I believe that will use > > libcryptopp if the dependency is installed on the build machine > > (last I looked). > > > > I forgot to mention the numbers below were based on v.91. > > > > Thanks, > > > > Stephen > > > > -----Original Message----- > > From: ceph-devel-owner@vger.kernel.org > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > > Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L > > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > > Subject: RE: wip-auth > > > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > > > I noticed that the spec file for building RPM's defaults to > > > building with libnss, instead of libcrypto++. Since the > > > measurements I'd done so far were from those RPM's I rebuilt with > > > libcrypto++.. so FWIW here is the difference between those two on > > > my system, memstore backend with a single OSD, and single > > > client. > > > > > > Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > > > Kernel: 3.10.0-123.13.2.el7 > > > > > > 100% 4K Writes, 1xOSD w/ Rados Bench > > > libnss | > > > Cryptopp # QD IOPS Latency(ms) | > > > IOPS Latency(ms) IOPS Improvement % 16 > > > 14432.57 1.11 | 18896.60 0.85 > > > 30.93% 100% 4K Reads, 1xOSD w/ Rados > > > Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) > > > IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% > > > > Yikes, 30%! I think this definitely worth some effort. We > > switched to libnss because it has the weird government > > certfiications that everyone wants and is more prevalent. crypto++ > > is also not packaged for Red Hat distros at all (presumably for > > that reason). > > > > I suspect that most of the overhead is in the encryption context > > setup and can be avoided with a bit of effort.. > > > > sage > > > > > > > > > > > > > Thanks, > > > > > > Stephen > > > > > > -----Original Message----- > > > From: ceph-devel-owner@vger.kernel.org > > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > > > Sent: Thursday, January 22, 2015 4:56 PM > > > To: andreas.bluemle@itxperts.de > > > Cc: ceph-devel@vger.kernel.org > > > Subject: wip-auth > > > > > > Hi Andreas, > > > > > > I took a look at the wip-auth I mentioned in the security call > > > last week... and the patch didn't work at all. Sorry if you > > > wasted any time trying it. > > > > > > Anyway, I fixed it up so that it actually worked and made one > > > other optimization. It would be great to hear what latencies you > > > measure with the changes in place. > > > > > > Also, it might be worth trying --with-cryptopp (or --with-nss if > > > you built cryptopp by default) to see if there is a difference. > > > There is a ton of boilerplate setting up encryption contexts and > > > key structures and so on that I suspect could be cached (perhaps > > > stashed in the CryptoKey struct?) with a bit of effort. See > > > > > > https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 > > > > > > sage > > > -- > > > To unsubscribe from this list: send the line "unsubscribe > > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > -- Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de ITXperts GmbH http://www.itxperts.de Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 Company details: http://www.itxperts.de/imprint.htm [-- Attachment #2: ceph-mon-0.91-wip-auth-backtrace.txt --] [-- Type: text/plain, Size: 1093 bytes --] 0> 2015-01-30 08:00:15.644464 7f3229507700 -1 *** Caught signal (Segmentation fault) ** in thread 7f3229507700 ceph version 0.91 (725d66098c98c2008b5fa07538325cc6816ca4a1) 1: /usr/bin/ceph-mon() [0x8ea1a5] 2: /lib64/libc.so.6() [0x316e4329a0] 3: (CryptoKey::encrypt(CephContext*, ceph::buffer::list const&, ceph::buffer::list&, std::string&) const+0x41) [0x6c2c51] 4: (void encode_encrypt_enc_bl<CephXServiceTicketInfo>(CephContext*, CephXServiceTicketInfo const&, CryptoKey const&, ceph::buffer::list&, std::string&)+0x26a) [0x6bf8ea] 5: (cephx_build_service_ticket_blob(CephContext*, CephXSessionAuthInfo&, CephXTicketBlob&)+0x2b2) [0x6bb002] 6: (Monitor::ms_get_authorizer(int, AuthAuthorizer**, bool)+0x39d) [0x57755d] 7: (SimpleMessenger::get_authorizer(int, bool)+0x4b) [0x8b3e5b] 8: (Pipe::connect()+0x1975) [0x8d8625] 9: (Pipe::writer()+0x9f3) [0x8db713] 10: (Pipe::Writer::entry()+0xd) [0x8de4ed] 11: /lib64/libpthread.so.0() [0x316ec079d1] 12: (clone()+0x6d) [0x316e4e8b7d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: wip-auth 2015-01-30 7:10 ` wip-auth Andreas Bluemle @ 2015-01-30 21:08 ` Sage Weil 2015-02-04 23:24 ` wip-auth Mark Nelson 0 siblings, 1 reply; 15+ messages in thread From: Sage Weil @ 2015-01-30 21:08 UTC (permalink / raw) To: Andreas Bluemle; +Cc: Blinick, Stephen L, ceph-devel Hi Andreas, It looks like that was a stale sha1, but the newer one was also broken. I've retested and it's working for me now. See latest wip-auth, sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. Thanks! sage On Fri, 30 Jan 2015, Andreas Bluemle wrote: > Hi Sage, > > I tried to integrate wip-auth into my 0.91 build environment. > > I had not been able to start the cluster successfully: ceph-mon > crashes with a segmentation fault in CryptoKey::encrypt() > (see attachment). > > This happens when linking with libnss or libcryptopp (version 5.6.2). > > I created the patch to add wip-auth based on github > pull request 3523 and was able to use this patch directly > with 0.91 with only a minor adaptation for common/ceph_context.h: > the 0.91 version of ceph_context.h did not know anything about > the experimental "class CephContextObs". > > wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. > > As my build environment is rpm, I had to modify the invocation > of the configure script in the spec file instead of the do_autogen.sh > script. > > > Best Regards > > Andreas Bluemle > > > On Tue, 27 Jan 2015 09:18:45 -0800 (PST) > Sage Weil <sweil@redhat.com> wrote: > > > I spent some time focusing on just CryptoKey::encrypt(). I > > benchmarked 100,000 encrypts of 128 bytes and got, at baseline, > > > > cryptopp: 100000 encoded in 0.655651 > > libnss : 100000 encoded in 1.288786 > > > > Ouch! With a (fixed) version of my earlier patch that avoids > > uselessly copying the input buffer: > > > > 100000 encoded in 1.231977 > > > > With a patch that puts the key structures in CryptoKey instead of > > recreating them each time: > > > > 100000 encoded in 0.396208 -- ~70% improvement over original > > > > This is pushed to wip-auth. There's also a patch that caches key > > structs for crypopp.. it now takes > > > > 100000 encoded in 0.440758 -- ~33% improvement over original > > > > (Not that almost anybody will ever care; we use libnss by default for > > both rpm and deb distros.) > > > > So, yay, nss is now a bit faster. What I'm not completely certain > > about is whether the structures I've preserved are truly stateless > > (and can be shared across threads, etc.). They encrypt/decrypt > > methods are const so, if the libraries are const-correct, it should > > be fine... but perhaps someone familiar with nss and/or crypto++ can > > review this? > > > > This is pushed to the latest wip-auth branch: > > > > https://github.com/ceph/ceph/commits/wip-auth > > > > Andreas and Stephen, what effect does this have on your numbers? > > > > Thanks! > > sage > > > > > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > > > > > Good to know, I was wondering why the spec file defaulted to > > > lib-nss.. the dpkg-build for debian packages just uses whatever > > > configuration you had built, and I believe that will use > > > libcryptopp if the dependency is installed on the build machine > > > (last I looked). > > > > > > I forgot to mention the numbers below were based on v.91. > > > > > > Thanks, > > > > > > Stephen > > > > > > -----Original Message----- > > > From: ceph-devel-owner@vger.kernel.org > > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > > > Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L > > > Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > > > Subject: RE: wip-auth > > > > > > On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > > > > I noticed that the spec file for building RPM's defaults to > > > > building with libnss, instead of libcrypto++. Since the > > > > measurements I'd done so far were from those RPM's I rebuilt with > > > > libcrypto++.. so FWIW here is the difference between those two on > > > > my system, memstore backend with a single OSD, and single > > > > client. > > > > > > > > Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > > > > Kernel: 3.10.0-123.13.2.el7 > > > > > > > > 100% 4K Writes, 1xOSD w/ Rados Bench > > > > libnss | > > > > Cryptopp # QD IOPS Latency(ms) | > > > > IOPS Latency(ms) IOPS Improvement % 16 > > > > 14432.57 1.11 | 18896.60 0.85 > > > > 30.93% 100% 4K Reads, 1xOSD w/ Rados > > > > Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) > > > > IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% > > > > > > Yikes, 30%! I think this definitely worth some effort. We > > > switched to libnss because it has the weird government > > > certfiications that everyone wants and is more prevalent. crypto++ > > > is also not packaged for Red Hat distros at all (presumably for > > > that reason). > > > > > > I suspect that most of the overhead is in the encryption context > > > setup and can be avoided with a bit of effort.. > > > > > > sage > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Stephen > > > > > > > > -----Original Message----- > > > > From: ceph-devel-owner@vger.kernel.org > > > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > > > > Sent: Thursday, January 22, 2015 4:56 PM > > > > To: andreas.bluemle@itxperts.de > > > > Cc: ceph-devel@vger.kernel.org > > > > Subject: wip-auth > > > > > > > > Hi Andreas, > > > > > > > > I took a look at the wip-auth I mentioned in the security call > > > > last week... and the patch didn't work at all. Sorry if you > > > > wasted any time trying it. > > > > > > > > Anyway, I fixed it up so that it actually worked and made one > > > > other optimization. It would be great to hear what latencies you > > > > measure with the changes in place. > > > > > > > > Also, it might be worth trying --with-cryptopp (or --with-nss if > > > > you built cryptopp by default) to see if there is a difference. > > > > There is a ton of boilerplate setting up encryption contexts and > > > > key structures and so on that I suspect could be cached (perhaps > > > > stashed in the CryptoKey struct?) with a bit of effort. See > > > > > > > > https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 > > > > > > > > sage > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe > > > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe > > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- To unsubscribe from this list: send the line "unsubscribe > > > ceph-devel" in the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > > -- > Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de > ITXperts GmbH http://www.itxperts.de > Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 > D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > > Company details: http://www.itxperts.de/imprint.htm > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: wip-auth 2015-01-30 21:08 ` wip-auth Sage Weil @ 2015-02-04 23:24 ` Mark Nelson [not found] ` <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv> 2015-03-10 20:54 ` wip-auth Blinick, Stephen L 0 siblings, 2 replies; 15+ messages in thread From: Mark Nelson @ 2015-02-04 23:24 UTC (permalink / raw) To: Sage Weil, Andreas Bluemle; +Cc: Blinick, Stephen L, ceph-devel Hi All, I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. Next up will be to see if how much this matters when testing against on SSDs. Here are the results: sync 4k object writes ===================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.99 1007 Wip-auth Ubuntu 14.04 Yes 0.81 1237 master Ubuntu 14.04 No 0.64 1549 Wip-auth Ubuntu 14.04 No 0.65 1563 master RHEL7 No 0.32 3158 sync 4k object reads ==================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.59 1695 Wip-auth Ubuntu 14.04 Yes 0.41 2409 master Ubuntu 14.04 No 0.29 3425 Wip-auth Ubuntu 14.04 No 0.29 3474 master RHEL7 No 0.17 5853 256 concurrent 4k object writes =============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 40.39 6339 Wip-auth Ubuntu 14.04 Yes 26.22 9763 master Ubuntu 14.04 No 17.46 14662 Wip-auth Ubuntu 14.04 No 17.34 14759 master RHEL7 No 14.93 17139 256 concurrent 4k object reads ============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 31.47 8134 Wip-auth Ubuntu 14.04 Yes 19.81 12922 master Ubuntu 14.04 No 12.82 19968 Wip-auth Ubuntu 14.04 No 12.75 20080 master RHEL7 No 12.04 21257 Mark On 01/30/2015 03:08 PM, Sage Weil wrote: > Hi Andreas, > > It looks like that was a stale sha1, but the newer one was also > broken. I've retested and it's working for me now. See latest wip-auth, > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. > > Thanks! > sage > > On Fri, 30 Jan 2015, Andreas Bluemle wrote: > >> Hi Sage, >> >> I tried to integrate wip-auth into my 0.91 build environment. >> >> I had not been able to start the cluster successfully: ceph-mon >> crashes with a segmentation fault in CryptoKey::encrypt() >> (see attachment). >> >> This happens when linking with libnss or libcryptopp (version 5.6.2). >> >> I created the patch to add wip-auth based on github >> pull request 3523 and was able to use this patch directly >> with 0.91 with only a minor adaptation for common/ceph_context.h: >> the 0.91 version of ceph_context.h did not know anything about >> the experimental "class CephContextObs". >> >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. >> >> As my build environment is rpm, I had to modify the invocation >> of the configure script in the spec file instead of the do_autogen.sh >> script. >> >> >> Best Regards >> >> Andreas Bluemle >> >> >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) >> Sage Weil <sweil@redhat.com> wrote: >> >>> I spent some time focusing on just CryptoKey::encrypt(). I >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, >>> >>> cryptopp: 100000 encoded in 0.655651 >>> libnss : 100000 encoded in 1.288786 >>> >>> Ouch! With a (fixed) version of my earlier patch that avoids >>> uselessly copying the input buffer: >>> >>> 100000 encoded in 1.231977 >>> >>> With a patch that puts the key structures in CryptoKey instead of >>> recreating them each time: >>> >>> 100000 encoded in 0.396208 -- ~70% improvement over original >>> >>> This is pushed to wip-auth. There's also a patch that caches key >>> structs for crypopp.. it now takes >>> >>> 100000 encoded in 0.440758 -- ~33% improvement over original >>> >>> (Not that almost anybody will ever care; we use libnss by default for >>> both rpm and deb distros.) >>> >>> So, yay, nss is now a bit faster. What I'm not completely certain >>> about is whether the structures I've preserved are truly stateless >>> (and can be shared across threads, etc.). They encrypt/decrypt >>> methods are const so, if the libraries are const-correct, it should >>> be fine... but perhaps someone familiar with nss and/or crypto++ can >>> review this? >>> >>> This is pushed to the latest wip-auth branch: >>> >>> https://github.com/ceph/ceph/commits/wip-auth >>> >>> Andreas and Stephen, what effect does this have on your numbers? >>> >>> Thanks! >>> sage >>> >>> >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>> >>>> Good to know, I was wondering why the spec file defaulted to >>>> lib-nss.. the dpkg-build for debian packages just uses whatever >>>> configuration you had built, and I believe that will use >>>> libcryptopp if the dependency is installed on the build machine >>>> (last I looked). >>>> >>>> I forgot to mention the numbers below were based on v.91. >>>> >>>> Thanks, >>>> >>>> Stephen >>>> >>>> -----Original Message----- >>>> From: ceph-devel-owner@vger.kernel.org >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org >>>> Subject: RE: wip-auth >>>> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>>> I noticed that the spec file for building RPM's defaults to >>>>> building with libnss, instead of libcrypto++. Since the >>>>> measurements I'd done so far were from those RPM's I rebuilt with >>>>> libcrypto++.. so FWIW here is the difference between those two on >>>>> my system, memstore backend with a single OSD, and single >>>>> client. >>>>> >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >>>>> Kernel: 3.10.0-123.13.2.el7 >>>>> >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench >>>>> libnss | >>>>> Cryptopp # QD IOPS Latency(ms) | >>>>> IOPS Latency(ms) IOPS Improvement % 16 >>>>> 14432.57 1.11 | 18896.60 0.85 >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados >>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) >>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% >>>> >>>> Yikes, 30%! I think this definitely worth some effort. We >>>> switched to libnss because it has the weird government >>>> certfiications that everyone wants and is more prevalent. crypto++ >>>> is also not packaged for Red Hat distros at all (presumably for >>>> that reason). >>>> >>>> I suspect that most of the overhead is in the encryption context >>>> setup and can be avoided with a bit of effort.. >>>> >>>> sage >>>> >>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Stephen >>>>> >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>> Sent: Thursday, January 22, 2015 4:56 PM >>>>> To: andreas.bluemle@itxperts.de >>>>> Cc: ceph-devel@vger.kernel.org >>>>> Subject: wip-auth >>>>> >>>>> Hi Andreas, >>>>> >>>>> I took a look at the wip-auth I mentioned in the security call >>>>> last week... and the patch didn't work at all. Sorry if you >>>>> wasted any time trying it. >>>>> >>>>> Anyway, I fixed it up so that it actually worked and made one >>>>> other optimization. It would be great to hear what latencies you >>>>> measure with the changes in place. >>>>> >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if >>>>> you built cryptopp by default) to see if there is a difference. >>>>> There is a ton of boilerplate setting up encryption contexts and >>>>> key structures and so on that I suspect could be cached (perhaps >>>>> stashed in the CryptoKey struct?) with a bit of effort. See >>>>> >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >>>>> >>>>> sage >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>> >>> >> >> >> >> -- >> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de >> ITXperts GmbH http://www.itxperts.de >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 >> >> Company details: http://www.itxperts.de/imprint.htm >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv>]
* Re: wip-auth [not found] ` <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv> @ 2015-02-06 7:45 ` Alexandre DERUMIER 2015-02-09 18:24 ` wip-auth Mark Nelson 0 siblings, 1 reply; 15+ messages in thread From: Alexandre DERUMIER @ 2015-02-06 7:45 UTC (permalink / raw) To: Mark Nelson; +Cc: Sage Weil, Andreas Bluemle, Stephen L Blinick, ceph-devel Hi Mark, do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge) Not sure it's related, but I have found a bug report on ubuntu, about irqbalance not working correctly on ubuntu 14.04 https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065 ----- Mail original ----- De: "Mark Nelson" <mark.nelson@inktank.com> À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de> Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org> Envoyé: Jeudi 5 Février 2015 00:24:42 Objet: Re: wip-auth Hi All, I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. Next up will be to see if how much this matters when testing against on SSDs. Here are the results: sync 4k object writes ===================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.99 1007 Wip-auth Ubuntu 14.04 Yes 0.81 1237 master Ubuntu 14.04 No 0.64 1549 Wip-auth Ubuntu 14.04 No 0.65 1563 master RHEL7 No 0.32 3158 sync 4k object reads ==================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.59 1695 Wip-auth Ubuntu 14.04 Yes 0.41 2409 master Ubuntu 14.04 No 0.29 3425 Wip-auth Ubuntu 14.04 No 0.29 3474 master RHEL7 No 0.17 5853 256 concurrent 4k object writes =============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 40.39 6339 Wip-auth Ubuntu 14.04 Yes 26.22 9763 master Ubuntu 14.04 No 17.46 14662 Wip-auth Ubuntu 14.04 No 17.34 14759 master RHEL7 No 14.93 17139 256 concurrent 4k object reads ============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 31.47 8134 Wip-auth Ubuntu 14.04 Yes 19.81 12922 master Ubuntu 14.04 No 12.82 19968 Wip-auth Ubuntu 14.04 No 12.75 20080 master RHEL7 No 12.04 21257 Mark On 01/30/2015 03:08 PM, Sage Weil wrote: > Hi Andreas, > > It looks like that was a stale sha1, but the newer one was also > broken. I've retested and it's working for me now. See latest wip-auth, > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. > > Thanks! > sage > > On Fri, 30 Jan 2015, Andreas Bluemle wrote: > >> Hi Sage, >> >> I tried to integrate wip-auth into my 0.91 build environment. >> >> I had not been able to start the cluster successfully: ceph-mon >> crashes with a segmentation fault in CryptoKey::encrypt() >> (see attachment). >> >> This happens when linking with libnss or libcryptopp (version 5.6.2). >> >> I created the patch to add wip-auth based on github >> pull request 3523 and was able to use this patch directly >> with 0.91 with only a minor adaptation for common/ceph_context.h: >> the 0.91 version of ceph_context.h did not know anything about >> the experimental "class CephContextObs". >> >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. >> >> As my build environment is rpm, I had to modify the invocation >> of the configure script in the spec file instead of the do_autogen.sh >> script. >> >> >> Best Regards >> >> Andreas Bluemle >> >> >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) >> Sage Weil <sweil@redhat.com> wrote: >> >>> I spent some time focusing on just CryptoKey::encrypt(). I >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, >>> >>> cryptopp: 100000 encoded in 0.655651 >>> libnss : 100000 encoded in 1.288786 >>> >>> Ouch! With a (fixed) version of my earlier patch that avoids >>> uselessly copying the input buffer: >>> >>> 100000 encoded in 1.231977 >>> >>> With a patch that puts the key structures in CryptoKey instead of >>> recreating them each time: >>> >>> 100000 encoded in 0.396208 -- ~70% improvement over original >>> >>> This is pushed to wip-auth. There's also a patch that caches key >>> structs for crypopp.. it now takes >>> >>> 100000 encoded in 0.440758 -- ~33% improvement over original >>> >>> (Not that almost anybody will ever care; we use libnss by default for >>> both rpm and deb distros.) >>> >>> So, yay, nss is now a bit faster. What I'm not completely certain >>> about is whether the structures I've preserved are truly stateless >>> (and can be shared across threads, etc.). They encrypt/decrypt >>> methods are const so, if the libraries are const-correct, it should >>> be fine... but perhaps someone familiar with nss and/or crypto++ can >>> review this? >>> >>> This is pushed to the latest wip-auth branch: >>> >>> https://github.com/ceph/ceph/commits/wip-auth >>> >>> Andreas and Stephen, what effect does this have on your numbers? >>> >>> Thanks! >>> sage >>> >>> >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>> >>>> Good to know, I was wondering why the spec file defaulted to >>>> lib-nss.. the dpkg-build for debian packages just uses whatever >>>> configuration you had built, and I believe that will use >>>> libcryptopp if the dependency is installed on the build machine >>>> (last I looked). >>>> >>>> I forgot to mention the numbers below were based on v.91. >>>> >>>> Thanks, >>>> >>>> Stephen >>>> >>>> -----Original Message----- >>>> From: ceph-devel-owner@vger.kernel.org >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org >>>> Subject: RE: wip-auth >>>> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>>> I noticed that the spec file for building RPM's defaults to >>>>> building with libnss, instead of libcrypto++. Since the >>>>> measurements I'd done so far were from those RPM's I rebuilt with >>>>> libcrypto++.. so FWIW here is the difference between those two on >>>>> my system, memstore backend with a single OSD, and single >>>>> client. >>>>> >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >>>>> Kernel: 3.10.0-123.13.2.el7 >>>>> >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench >>>>> libnss | >>>>> Cryptopp # QD IOPS Latency(ms) | >>>>> IOPS Latency(ms) IOPS Improvement % 16 >>>>> 14432.57 1.11 | 18896.60 0.85 >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados >>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) >>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% >>>> >>>> Yikes, 30%! I think this definitely worth some effort. We >>>> switched to libnss because it has the weird government >>>> certfiications that everyone wants and is more prevalent. crypto++ >>>> is also not packaged for Red Hat distros at all (presumably for >>>> that reason). >>>> >>>> I suspect that most of the overhead is in the encryption context >>>> setup and can be avoided with a bit of effort.. >>>> >>>> sage >>>> >>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Stephen >>>>> >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>> Sent: Thursday, January 22, 2015 4:56 PM >>>>> To: andreas.bluemle@itxperts.de >>>>> Cc: ceph-devel@vger.kernel.org >>>>> Subject: wip-auth >>>>> >>>>> Hi Andreas, >>>>> >>>>> I took a look at the wip-auth I mentioned in the security call >>>>> last week... and the patch didn't work at all. Sorry if you >>>>> wasted any time trying it. >>>>> >>>>> Anyway, I fixed it up so that it actually worked and made one >>>>> other optimization. It would be great to hear what latencies you >>>>> measure with the changes in place. >>>>> >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if >>>>> you built cryptopp by default) to see if there is a difference. >>>>> There is a ton of boilerplate setting up encryption contexts and >>>>> key structures and so on that I suspect could be cached (perhaps >>>>> stashed in the CryptoKey struct?) with a bit of effort. See >>>>> >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >>>>> >>>>> sage >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>> >>> >> >> >> >> -- >> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de >> ITXperts GmbH http://www.itxperts.de >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 >> >> Company details: http://www.itxperts.de/imprint.htm >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: wip-auth 2015-02-06 7:45 ` wip-auth Alexandre DERUMIER @ 2015-02-09 18:24 ` Mark Nelson [not found] ` <987318816.223626.1423542893540.JavaMail.zimbra@oxygem.tv> 0 siblings, 1 reply; 15+ messages in thread From: Mark Nelson @ 2015-02-09 18:24 UTC (permalink / raw) To: Alexandre DERUMIER Cc: Sage Weil, Andreas Bluemle, Stephen L Blinick, ceph-devel Hi Alex, So far we don't really know, but suspect kernel/networking related things. sysctl -a showed some differences in various vm/numa/ipv4 settings. Simply changing settings around might be a good first step, but if it's kernel related that might be tougher to track down. The first step was to make sure it wasn't due to auth as that was suspect as well, but it seems to be an independent issue. Mark On 02/06/2015 01:45 AM, Alexandre DERUMIER wrote: > Hi Mark, > > do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge) > > > Not sure it's related, but I have found a bug report on ubuntu, > about irqbalance not working correctly on ubuntu 14.04 > https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065 > > > ----- Mail original ----- > De: "Mark Nelson" <mark.nelson@inktank.com> > À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de> > Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org> > Envoyé: Jeudi 5 Février 2015 00:24:42 > Objet: Re: wip-auth > > Hi All, > > I completed some tests with wip-auth to the memstore on ubuntu earlier > today. Basic gist of it is that the improvements in wip-auth help but > don't quite get us to what can be achieved with auth disabled. RHEL7 > (without auth) continues to do very well in latency bound situations. > Next up will be to see if how much this matters when testing against on > SSDs. > > Here are the results: > > sync 4k object writes > ===================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.99 1007 > Wip-auth Ubuntu 14.04 Yes 0.81 1237 > master Ubuntu 14.04 No 0.64 1549 > Wip-auth Ubuntu 14.04 No 0.65 1563 > master RHEL7 No 0.32 3158 > > sync 4k object reads > ==================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.59 1695 > Wip-auth Ubuntu 14.04 Yes 0.41 2409 > master Ubuntu 14.04 No 0.29 3425 > Wip-auth Ubuntu 14.04 No 0.29 3474 > master RHEL7 No 0.17 5853 > > 256 concurrent 4k object writes > =============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 40.39 6339 > Wip-auth Ubuntu 14.04 Yes 26.22 9763 > master Ubuntu 14.04 No 17.46 14662 > Wip-auth Ubuntu 14.04 No 17.34 14759 > master RHEL7 No 14.93 17139 > > 256 concurrent 4k object reads > ============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 31.47 8134 > Wip-auth Ubuntu 14.04 Yes 19.81 12922 > master Ubuntu 14.04 No 12.82 19968 > Wip-auth Ubuntu 14.04 No 12.75 20080 > master RHEL7 No 12.04 21257 > > Mark > > On 01/30/2015 03:08 PM, Sage Weil wrote: >> Hi Andreas, >> >> It looks like that was a stale sha1, but the newer one was also >> broken. I've retested and it's working for me now. See latest wip-auth, >> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. >> >> Thanks! >> sage >> >> On Fri, 30 Jan 2015, Andreas Bluemle wrote: >> >>> Hi Sage, >>> >>> I tried to integrate wip-auth into my 0.91 build environment. >>> >>> I had not been able to start the cluster successfully: ceph-mon >>> crashes with a segmentation fault in CryptoKey::encrypt() >>> (see attachment). >>> >>> This happens when linking with libnss or libcryptopp (version 5.6.2). >>> >>> I created the patch to add wip-auth based on github >>> pull request 3523 and was able to use this patch directly >>> with 0.91 with only a minor adaptation for common/ceph_context.h: >>> the 0.91 version of ceph_context.h did not know anything about >>> the experimental "class CephContextObs". >>> >>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. >>> >>> As my build environment is rpm, I had to modify the invocation >>> of the configure script in the spec file instead of the do_autogen.sh >>> script. >>> >>> >>> Best Regards >>> >>> Andreas Bluemle >>> >>> >>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) >>> Sage Weil <sweil@redhat.com> wrote: >>> >>>> I spent some time focusing on just CryptoKey::encrypt(). I >>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, >>>> >>>> cryptopp: 100000 encoded in 0.655651 >>>> libnss : 100000 encoded in 1.288786 >>>> >>>> Ouch! With a (fixed) version of my earlier patch that avoids >>>> uselessly copying the input buffer: >>>> >>>> 100000 encoded in 1.231977 >>>> >>>> With a patch that puts the key structures in CryptoKey instead of >>>> recreating them each time: >>>> >>>> 100000 encoded in 0.396208 -- ~70% improvement over original >>>> >>>> This is pushed to wip-auth. There's also a patch that caches key >>>> structs for crypopp.. it now takes >>>> >>>> 100000 encoded in 0.440758 -- ~33% improvement over original >>>> >>>> (Not that almost anybody will ever care; we use libnss by default for >>>> both rpm and deb distros.) >>>> >>>> So, yay, nss is now a bit faster. What I'm not completely certain >>>> about is whether the structures I've preserved are truly stateless >>>> (and can be shared across threads, etc.). They encrypt/decrypt >>>> methods are const so, if the libraries are const-correct, it should >>>> be fine... but perhaps someone familiar with nss and/or crypto++ can >>>> review this? >>>> >>>> This is pushed to the latest wip-auth branch: >>>> >>>> https://github.com/ceph/ceph/commits/wip-auth >>>> >>>> Andreas and Stephen, what effect does this have on your numbers? >>>> >>>> Thanks! >>>> sage >>>> >>>> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>> >>>>> Good to know, I was wondering why the spec file defaulted to >>>>> lib-nss.. the dpkg-build for debian packages just uses whatever >>>>> configuration you had built, and I believe that will use >>>>> libcryptopp if the dependency is installed on the build machine >>>>> (last I looked). >>>>> >>>>> I forgot to mention the numbers below were based on v.91. >>>>> >>>>> Thanks, >>>>> >>>>> Stephen >>>>> >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L >>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org >>>>> Subject: RE: wip-auth >>>>> >>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>>>> I noticed that the spec file for building RPM's defaults to >>>>>> building with libnss, instead of libcrypto++. Since the >>>>>> measurements I'd done so far were from those RPM's I rebuilt with >>>>>> libcrypto++.. so FWIW here is the difference between those two on >>>>>> my system, memstore backend with a single OSD, and single >>>>>> client. >>>>>> >>>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >>>>>> Kernel: 3.10.0-123.13.2.el7 >>>>>> >>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench >>>>>> libnss | >>>>>> Cryptopp # QD IOPS Latency(ms) | >>>>>> IOPS Latency(ms) IOPS Improvement % 16 >>>>>> 14432.57 1.11 | 18896.60 0.85 >>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados >>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) >>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% >>>>> >>>>> Yikes, 30%! I think this definitely worth some effort. We >>>>> switched to libnss because it has the weird government >>>>> certfiications that everyone wants and is more prevalent. crypto++ >>>>> is also not packaged for Red Hat distros at all (presumably for >>>>> that reason). >>>>> >>>>> I suspect that most of the overhead is in the encryption context >>>>> setup and can be avoided with a bit of effort.. >>>>> >>>>> sage >>>>> >>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Stephen >>>>>> >>>>>> -----Original Message----- >>>>>> From: ceph-devel-owner@vger.kernel.org >>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>>> Sent: Thursday, January 22, 2015 4:56 PM >>>>>> To: andreas.bluemle@itxperts.de >>>>>> Cc: ceph-devel@vger.kernel.org >>>>>> Subject: wip-auth >>>>>> >>>>>> Hi Andreas, >>>>>> >>>>>> I took a look at the wip-auth I mentioned in the security call >>>>>> last week... and the patch didn't work at all. Sorry if you >>>>>> wasted any time trying it. >>>>>> >>>>>> Anyway, I fixed it up so that it actually worked and made one >>>>>> other optimization. It would be great to hear what latencies you >>>>>> measure with the changes in place. >>>>>> >>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if >>>>>> you built cryptopp by default) to see if there is a difference. >>>>>> There is a ton of boilerplate setting up encryption contexts and >>>>>> key structures and so on that I suspect could be cached (perhaps >>>>>> stashed in the CryptoKey struct?) with a bit of effort. See >>>>>> >>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >>>>>> >>>>>> sage >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> -- To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de >>> ITXperts GmbH http://www.itxperts.de >>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 >>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 >>> >>> Company details: http://www.itxperts.de/imprint.htm >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <987318816.223626.1423542893540.JavaMail.zimbra@oxygem.tv>]
* Re: wip-auth [not found] ` <987318816.223626.1423542893540.JavaMail.zimbra@oxygem.tv> @ 2015-02-10 4:36 ` Alexandre DERUMIER 0 siblings, 0 replies; 15+ messages in thread From: Alexandre DERUMIER @ 2015-02-10 4:36 UTC (permalink / raw) To: Mark Nelson; +Cc: Sage Weil, Andreas Bluemle, Stephen L Blinick, ceph-devel >>o far we don't really know, but suspect kernel/networking related >>things. sysctl -a showed some differences in various vm/numa/ipv4 >>settings. Simply changing settings around might be a good first step, >>but if it's kernel related that might be tougher to track down. The >>first step was to make sure it wasn't due to auth as that was suspect as >>well, but it seems to be an independent issue. Ok thanks. I'll try to compare too next month between debian and centos ----- Mail original ----- De: "Mark Nelson" <mnelson@redhat.com> À: "aderumier" <aderumier@odiso.com> Cc: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de>, "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org> Envoyé: Lundi 9 Février 2015 19:24:18 Objet: Re: wip-auth Hi Alex, So far we don't really know, but suspect kernel/networking related things. sysctl -a showed some differences in various vm/numa/ipv4 settings. Simply changing settings around might be a good first step, but if it's kernel related that might be tougher to track down. The first step was to make sure it wasn't due to auth as that was suspect as well, but it seems to be an independent issue. Mark On 02/06/2015 01:45 AM, Alexandre DERUMIER wrote: > Hi Mark, > > do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge) > > > Not sure it's related, but I have found a bug report on ubuntu, > about irqbalance not working correctly on ubuntu 14.04 > https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065 > > > ----- Mail original ----- > De: "Mark Nelson" <mark.nelson@inktank.com> > À: "Sage Weil" <sweil@redhat.com>, "Andreas Bluemle" <andreas.bluemle@itxperts.de> > Cc: "Stephen L Blinick" <stephen.l.blinick@intel.com>, "ceph-devel" <ceph-devel@vger.kernel.org> > Envoyé: Jeudi 5 Février 2015 00:24:42 > Objet: Re: wip-auth > > Hi All, > > I completed some tests with wip-auth to the memstore on ubuntu earlier > today. Basic gist of it is that the improvements in wip-auth help but > don't quite get us to what can be achieved with auth disabled. RHEL7 > (without auth) continues to do very well in latency bound situations. > Next up will be to see if how much this matters when testing against on > SSDs. > > Here are the results: > > sync 4k object writes > ===================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.99 1007 > Wip-auth Ubuntu 14.04 Yes 0.81 1237 > master Ubuntu 14.04 No 0.64 1549 > Wip-auth Ubuntu 14.04 No 0.65 1563 > master RHEL7 No 0.32 3158 > > sync 4k object reads > ==================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.59 1695 > Wip-auth Ubuntu 14.04 Yes 0.41 2409 > master Ubuntu 14.04 No 0.29 3425 > Wip-auth Ubuntu 14.04 No 0.29 3474 > master RHEL7 No 0.17 5853 > > 256 concurrent 4k object writes > =============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 40.39 6339 > Wip-auth Ubuntu 14.04 Yes 26.22 9763 > master Ubuntu 14.04 No 17.46 14662 > Wip-auth Ubuntu 14.04 No 17.34 14759 > master RHEL7 No 14.93 17139 > > 256 concurrent 4k object reads > ============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 31.47 8134 > Wip-auth Ubuntu 14.04 Yes 19.81 12922 > master Ubuntu 14.04 No 12.82 19968 > Wip-auth Ubuntu 14.04 No 12.75 20080 > master RHEL7 No 12.04 21257 > > Mark > > On 01/30/2015 03:08 PM, Sage Weil wrote: >> Hi Andreas, >> >> It looks like that was a stale sha1, but the newer one was also >> broken. I've retested and it's working for me now. See latest wip-auth, >> sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. >> >> Thanks! >> sage >> >> On Fri, 30 Jan 2015, Andreas Bluemle wrote: >> >>> Hi Sage, >>> >>> I tried to integrate wip-auth into my 0.91 build environment. >>> >>> I had not been able to start the cluster successfully: ceph-mon >>> crashes with a segmentation fault in CryptoKey::encrypt() >>> (see attachment). >>> >>> This happens when linking with libnss or libcryptopp (version 5.6.2). >>> >>> I created the patch to add wip-auth based on github >>> pull request 3523 and was able to use this patch directly >>> with 0.91 with only a minor adaptation for common/ceph_context.h: >>> the 0.91 version of ceph_context.h did not know anything about >>> the experimental "class CephContextObs". >>> >>> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. >>> >>> As my build environment is rpm, I had to modify the invocation >>> of the configure script in the spec file instead of the do_autogen.sh >>> script. >>> >>> >>> Best Regards >>> >>> Andreas Bluemle >>> >>> >>> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) >>> Sage Weil <sweil@redhat.com> wrote: >>> >>>> I spent some time focusing on just CryptoKey::encrypt(). I >>>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, >>>> >>>> cryptopp: 100000 encoded in 0.655651 >>>> libnss : 100000 encoded in 1.288786 >>>> >>>> Ouch! With a (fixed) version of my earlier patch that avoids >>>> uselessly copying the input buffer: >>>> >>>> 100000 encoded in 1.231977 >>>> >>>> With a patch that puts the key structures in CryptoKey instead of >>>> recreating them each time: >>>> >>>> 100000 encoded in 0.396208 -- ~70% improvement over original >>>> >>>> This is pushed to wip-auth. There's also a patch that caches key >>>> structs for crypopp.. it now takes >>>> >>>> 100000 encoded in 0.440758 -- ~33% improvement over original >>>> >>>> (Not that almost anybody will ever care; we use libnss by default for >>>> both rpm and deb distros.) >>>> >>>> So, yay, nss is now a bit faster. What I'm not completely certain >>>> about is whether the structures I've preserved are truly stateless >>>> (and can be shared across threads, etc.). They encrypt/decrypt >>>> methods are const so, if the libraries are const-correct, it should >>>> be fine... but perhaps someone familiar with nss and/or crypto++ can >>>> review this? >>>> >>>> This is pushed to the latest wip-auth branch: >>>> >>>> https://github.com/ceph/ceph/commits/wip-auth >>>> >>>> Andreas and Stephen, what effect does this have on your numbers? >>>> >>>> Thanks! >>>> sage >>>> >>>> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>> >>>>> Good to know, I was wondering why the spec file defaulted to >>>>> lib-nss.. the dpkg-build for debian packages just uses whatever >>>>> configuration you had built, and I believe that will use >>>>> libcryptopp if the dependency is installed on the build machine >>>>> (last I looked). >>>>> >>>>> I forgot to mention the numbers below were based on v.91. >>>>> >>>>> Thanks, >>>>> >>>>> Stephen >>>>> >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L >>>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org >>>>> Subject: RE: wip-auth >>>>> >>>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>>>> I noticed that the spec file for building RPM's defaults to >>>>>> building with libnss, instead of libcrypto++. Since the >>>>>> measurements I'd done so far were from those RPM's I rebuilt with >>>>>> libcrypto++.. so FWIW here is the difference between those two on >>>>>> my system, memstore backend with a single OSD, and single >>>>>> client. >>>>>> >>>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >>>>>> Kernel: 3.10.0-123.13.2.el7 >>>>>> >>>>>> 100% 4K Writes, 1xOSD w/ Rados Bench >>>>>> libnss | >>>>>> Cryptopp # QD IOPS Latency(ms) | >>>>>> IOPS Latency(ms) IOPS Improvement % 16 >>>>>> 14432.57 1.11 | 18896.60 0.85 >>>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados >>>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms) >>>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62% >>>>> >>>>> Yikes, 30%! I think this definitely worth some effort. We >>>>> switched to libnss because it has the weird government >>>>> certfiications that everyone wants and is more prevalent. crypto++ >>>>> is also not packaged for Red Hat distros at all (presumably for >>>>> that reason). >>>>> >>>>> I suspect that most of the overhead is in the encryption context >>>>> setup and can be avoided with a bit of effort.. >>>>> >>>>> sage >>>>> >>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Stephen >>>>>> >>>>>> -----Original Message----- >>>>>> From: ceph-devel-owner@vger.kernel.org >>>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>>> Sent: Thursday, January 22, 2015 4:56 PM >>>>>> To: andreas.bluemle@itxperts.de >>>>>> Cc: ceph-devel@vger.kernel.org >>>>>> Subject: wip-auth >>>>>> >>>>>> Hi Andreas, >>>>>> >>>>>> I took a look at the wip-auth I mentioned in the security call >>>>>> last week... and the patch didn't work at all. Sorry if you >>>>>> wasted any time trying it. >>>>>> >>>>>> Anyway, I fixed it up so that it actually worked and made one >>>>>> other optimization. It would be great to hear what latencies you >>>>>> measure with the changes in place. >>>>>> >>>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if >>>>>> you built cryptopp by default) to see if there is a difference. >>>>>> There is a ton of boilerplate setting up encryption contexts and >>>>>> key structures and so on that I suspect could be cached (perhaps >>>>>> stashed in the CryptoKey struct?) with a bit of effort. See >>>>>> >>>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213 >>>>>> >>>>>> sage >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> -- To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de >>> ITXperts GmbH http://www.itxperts.de >>> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 >>> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 >>> >>> Company details: http://www.itxperts.de/imprint.htm >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-02-04 23:24 ` wip-auth Mark Nelson [not found] ` <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv> @ 2015-03-10 20:54 ` Blinick, Stephen L 2015-03-10 20:55 ` wip-auth Sage Weil 1 sibling, 1 reply; 15+ messages in thread From: Blinick, Stephen L @ 2015-03-10 20:54 UTC (permalink / raw) To: mnelson, Sage Weil; +Cc: ceph-devel Hi -- was running some baselines on 0.93 and wanted to check, will wip-auth changes go into Hamer or in a later release? (Sorry if I missed the discussion in one of the perf meetings last month) Thanks, Stephen -----Original Message----- From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson Sent: Wednesday, February 4, 2015 4:25 PM To: Sage Weil; Andreas Bluemle Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org Subject: Re: wip-auth Hi All, I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. Next up will be to see if how much this matters when testing against on SSDs. Here are the results: sync 4k object writes ===================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.99 1007 Wip-auth Ubuntu 14.04 Yes 0.81 1237 master Ubuntu 14.04 No 0.64 1549 Wip-auth Ubuntu 14.04 No 0.65 1563 master RHEL7 No 0.32 3158 sync 4k object reads ==================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 0.59 1695 Wip-auth Ubuntu 14.04 Yes 0.41 2409 master Ubuntu 14.04 No 0.29 3425 Wip-auth Ubuntu 14.04 No 0.29 3474 master RHEL7 No 0.17 5853 256 concurrent 4k object writes =============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 40.39 6339 Wip-auth Ubuntu 14.04 Yes 26.22 9763 master Ubuntu 14.04 No 17.46 14662 Wip-auth Ubuntu 14.04 No 17.34 14759 master RHEL7 No 14.93 17139 256 concurrent 4k object reads ============================== Ceph OS Auth Avg Lat (ms) Avg IOPS ---------------------------------------------------------------- master Ubuntu 14.04 Yes 31.47 8134 Wip-auth Ubuntu 14.04 Yes 19.81 12922 master Ubuntu 14.04 No 12.82 19968 Wip-auth Ubuntu 14.04 No 12.75 20080 master RHEL7 No 12.04 21257 Mark On 01/30/2015 03:08 PM, Sage Weil wrote: > Hi Andreas, > > It looks like that was a stale sha1, but the newer one was also > broken. I've retested and it's working for me now. See latest > wip-auth, > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. > > Thanks! > sage > > On Fri, 30 Jan 2015, Andreas Bluemle wrote: > >> Hi Sage, >> >> I tried to integrate wip-auth into my 0.91 build environment. >> >> I had not been able to start the cluster successfully: ceph-mon >> crashes with a segmentation fault in CryptoKey::encrypt() (see >> attachment). >> >> This happens when linking with libnss or libcryptopp (version 5.6.2). >> >> I created the patch to add wip-auth based on github pull request 3523 >> and was able to use this patch directly with 0.91 with only a minor >> adaptation for common/ceph_context.h: >> the 0.91 version of ceph_context.h did not know anything about the >> experimental "class CephContextObs". >> >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. >> >> As my build environment is rpm, I had to modify the invocation of the >> configure script in the spec file instead of the do_autogen.sh >> script. >> >> >> Best Regards >> >> Andreas Bluemle >> >> >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@redhat.com> >> wrote: >> >>> I spent some time focusing on just CryptoKey::encrypt(). I >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, >>> >>> cryptopp: 100000 encoded in 0.655651 >>> libnss : 100000 encoded in 1.288786 >>> >>> Ouch! With a (fixed) version of my earlier patch that avoids >>> uselessly copying the input buffer: >>> >>> 100000 encoded in 1.231977 >>> >>> With a patch that puts the key structures in CryptoKey instead of >>> recreating them each time: >>> >>> 100000 encoded in 0.396208 -- ~70% improvement over original >>> >>> This is pushed to wip-auth. There's also a patch that caches key >>> structs for crypopp.. it now takes >>> >>> 100000 encoded in 0.440758 -- ~33% improvement over original >>> >>> (Not that almost anybody will ever care; we use libnss by default >>> for both rpm and deb distros.) >>> >>> So, yay, nss is now a bit faster. What I'm not completely certain >>> about is whether the structures I've preserved are truly stateless >>> (and can be shared across threads, etc.). They encrypt/decrypt >>> methods are const so, if the libraries are const-correct, it should >>> be fine... but perhaps someone familiar with nss and/or crypto++ can >>> review this? >>> >>> This is pushed to the latest wip-auth branch: >>> >>> https://github.com/ceph/ceph/commits/wip-auth >>> >>> Andreas and Stephen, what effect does this have on your numbers? >>> >>> Thanks! >>> sage >>> >>> >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>> >>>> Good to know, I was wondering why the spec file defaulted to >>>> lib-nss.. the dpkg-build for debian packages just uses whatever >>>> configuration you had built, and I believe that will use >>>> libcryptopp if the dependency is installed on the build machine >>>> (last I looked). >>>> >>>> I forgot to mention the numbers below were based on v.91. >>>> >>>> Thanks, >>>> >>>> Stephen >>>> >>>> -----Original Message----- >>>> From: ceph-devel-owner@vger.kernel.org >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org >>>> Subject: RE: wip-auth >>>> >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: >>>>> I noticed that the spec file for building RPM's defaults to >>>>> building with libnss, instead of libcrypto++. Since the >>>>> measurements I'd done so far were from those RPM's I rebuilt with >>>>> libcrypto++.. so FWIW here is the difference between those two on >>>>> my system, memstore backend with a single OSD, and single client. >>>>> >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 >>>>> Kernel: 3.10.0-123.13.2.el7 >>>>> >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench >>>>> libnss | >>>>> Cryptopp # QD IOPS Latency(ms) | >>>>> IOPS Latency(ms) IOPS Improvement % 16 >>>>> 14432.57 1.11 | 18896.60 0.85 >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD >>>>> IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % 16 >>>>> 19532.53 0.82 | 25708.70 0.62 31.62% >>>> >>>> Yikes, 30%! I think this definitely worth some effort. We >>>> switched to libnss because it has the weird government >>>> certfiications that everyone wants and is more prevalent. crypto++ >>>> is also not packaged for Red Hat distros at all (presumably for >>>> that reason). >>>> >>>> I suspect that most of the overhead is in the encryption context >>>> setup and can be avoided with a bit of effort.. >>>> >>>> sage >>>> >>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Stephen >>>>> >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil >>>>> Sent: Thursday, January 22, 2015 4:56 PM >>>>> To: andreas.bluemle@itxperts.de >>>>> Cc: ceph-devel@vger.kernel.org >>>>> Subject: wip-auth >>>>> >>>>> Hi Andreas, >>>>> >>>>> I took a look at the wip-auth I mentioned in the security call >>>>> last week... and the patch didn't work at all. Sorry if you >>>>> wasted any time trying it. >>>>> >>>>> Anyway, I fixed it up so that it actually worked and made one >>>>> other optimization. It would be great to hear what latencies you >>>>> measure with the changes in place. >>>>> >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if >>>>> you built cryptopp by default) to see if there is a difference. >>>>> There is a ton of boilerplate setting up encryption contexts and >>>>> key structures and so on that I suspect could be cached (perhaps >>>>> stashed in the CryptoKey struct?) with a bit of effort. See >>>>> >>>>> >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L2 >>>>> 13 >>>>> >>>>> sage >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>> >>> >> >> >> >> -- >> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de >> ITXperts GmbH http://www.itxperts.de >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 >> >> Company details: http://www.itxperts.de/imprint.htm >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: wip-auth 2015-03-10 20:54 ` wip-auth Blinick, Stephen L @ 2015-03-10 20:55 ` Sage Weil 0 siblings, 0 replies; 15+ messages in thread From: Sage Weil @ 2015-03-10 20:55 UTC (permalink / raw) To: Blinick, Stephen L; +Cc: mnelson, ceph-devel On Tue, 10 Mar 2015, Blinick, Stephen L wrote: > Hi -- was running some baselines on 0.93 and wanted to check, will > wip-auth changes go into Hamer or in a later release? (Sorry if I > missed the discussion in one of the perf meetings last month) It will go in after hammer. We can backport as needed. sage > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson > Sent: Wednesday, February 4, 2015 4:25 PM > To: Sage Weil; Andreas Bluemle > Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org > Subject: Re: wip-auth > > Hi All, > > I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. > Next up will be to see if how much this matters when testing against on SSDs. > > Here are the results: > > sync 4k object writes > ===================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.99 1007 > Wip-auth Ubuntu 14.04 Yes 0.81 1237 > master Ubuntu 14.04 No 0.64 1549 > Wip-auth Ubuntu 14.04 No 0.65 1563 > master RHEL7 No 0.32 3158 > > sync 4k object reads > ==================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.59 1695 > Wip-auth Ubuntu 14.04 Yes 0.41 2409 > master Ubuntu 14.04 No 0.29 3425 > Wip-auth Ubuntu 14.04 No 0.29 3474 > master RHEL7 No 0.17 5853 > > 256 concurrent 4k object writes > =============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 40.39 6339 > Wip-auth Ubuntu 14.04 Yes 26.22 9763 > master Ubuntu 14.04 No 17.46 14662 > Wip-auth Ubuntu 14.04 No 17.34 14759 > master RHEL7 No 14.93 17139 > > 256 concurrent 4k object reads > ============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 31.47 8134 > Wip-auth Ubuntu 14.04 Yes 19.81 12922 > master Ubuntu 14.04 No 12.82 19968 > Wip-auth Ubuntu 14.04 No 12.75 20080 > master RHEL7 No 12.04 21257 > > Mark > > On 01/30/2015 03:08 PM, Sage Weil wrote: > > Hi Andreas, > > > > It looks like that was a stale sha1, but the newer one was also > > broken. I've retested and it's working for me now. See latest > > wip-auth, > > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. > > > > Thanks! > > sage > > > > On Fri, 30 Jan 2015, Andreas Bluemle wrote: > > > >> Hi Sage, > >> > >> I tried to integrate wip-auth into my 0.91 build environment. > >> > >> I had not been able to start the cluster successfully: ceph-mon > >> crashes with a segmentation fault in CryptoKey::encrypt() (see > >> attachment). > >> > >> This happens when linking with libnss or libcryptopp (version 5.6.2). > >> > >> I created the patch to add wip-auth based on github pull request 3523 > >> and was able to use this patch directly with 0.91 with only a minor > >> adaptation for common/ceph_context.h: > >> the 0.91 version of ceph_context.h did not know anything about the > >> experimental "class CephContextObs". > >> > >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. > >> > >> As my build environment is rpm, I had to modify the invocation of the > >> configure script in the spec file instead of the do_autogen.sh > >> script. > >> > >> > >> Best Regards > >> > >> Andreas Bluemle > >> > >> > >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@redhat.com> > >> wrote: > >> > >>> I spent some time focusing on just CryptoKey::encrypt(). I > >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, > >>> > >>> cryptopp: 100000 encoded in 0.655651 > >>> libnss : 100000 encoded in 1.288786 > >>> > >>> Ouch! With a (fixed) version of my earlier patch that avoids > >>> uselessly copying the input buffer: > >>> > >>> 100000 encoded in 1.231977 > >>> > >>> With a patch that puts the key structures in CryptoKey instead of > >>> recreating them each time: > >>> > >>> 100000 encoded in 0.396208 -- ~70% improvement over original > >>> > >>> This is pushed to wip-auth. There's also a patch that caches key > >>> structs for crypopp.. it now takes > >>> > >>> 100000 encoded in 0.440758 -- ~33% improvement over original > >>> > >>> (Not that almost anybody will ever care; we use libnss by default > >>> for both rpm and deb distros.) > >>> > >>> So, yay, nss is now a bit faster. What I'm not completely certain > >>> about is whether the structures I've preserved are truly stateless > >>> (and can be shared across threads, etc.). They encrypt/decrypt > >>> methods are const so, if the libraries are const-correct, it should > >>> be fine... but perhaps someone familiar with nss and/or crypto++ can > >>> review this? > >>> > >>> This is pushed to the latest wip-auth branch: > >>> > >>> https://github.com/ceph/ceph/commits/wip-auth > >>> > >>> Andreas and Stephen, what effect does this have on your numbers? > >>> > >>> Thanks! > >>> sage > >>> > >>> > >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > >>> > >>>> Good to know, I was wondering why the spec file defaulted to > >>>> lib-nss.. the dpkg-build for debian packages just uses whatever > >>>> configuration you had built, and I believe that will use > >>>> libcryptopp if the dependency is installed on the build machine > >>>> (last I looked). > >>>> > >>>> I forgot to mention the numbers below were based on v.91. > >>>> > >>>> Thanks, > >>>> > >>>> Stephen > >>>> > >>>> -----Original Message----- > >>>> From: ceph-devel-owner@vger.kernel.org > >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L > >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org > >>>> Subject: RE: wip-auth > >>>> > >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > >>>>> I noticed that the spec file for building RPM's defaults to > >>>>> building with libnss, instead of libcrypto++. Since the > >>>>> measurements I'd done so far were from those RPM's I rebuilt with > >>>>> libcrypto++.. so FWIW here is the difference between those two on > >>>>> my system, memstore backend with a single OSD, and single client. > >>>>> > >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > >>>>> Kernel: 3.10.0-123.13.2.el7 > >>>>> > >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench > >>>>> libnss | > >>>>> Cryptopp # QD IOPS Latency(ms) | > >>>>> IOPS Latency(ms) IOPS Improvement % 16 > >>>>> 14432.57 1.11 | 18896.60 0.85 > >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD > >>>>> IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % 16 > >>>>> 19532.53 0.82 | 25708.70 0.62 31.62% > >>>> > >>>> Yikes, 30%! I think this definitely worth some effort. We > >>>> switched to libnss because it has the weird government > >>>> certfiications that everyone wants and is more prevalent. crypto++ > >>>> is also not packaged for Red Hat distros at all (presumably for > >>>> that reason). > >>>> > >>>> I suspect that most of the overhead is in the encryption context > >>>> setup and can be avoided with a bit of effort.. > >>>> > >>>> sage > >>>> > >>>> > >>>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Stephen > >>>>> > >>>>> -----Original Message----- > >>>>> From: ceph-devel-owner@vger.kernel.org > >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil > >>>>> Sent: Thursday, January 22, 2015 4:56 PM > >>>>> To: andreas.bluemle@itxperts.de > >>>>> Cc: ceph-devel@vger.kernel.org > >>>>> Subject: wip-auth > >>>>> > >>>>> Hi Andreas, > >>>>> > >>>>> I took a look at the wip-auth I mentioned in the security call > >>>>> last week... and the patch didn't work at all. Sorry if you > >>>>> wasted any time trying it. > >>>>> > >>>>> Anyway, I fixed it up so that it actually worked and made one > >>>>> other optimization. It would be great to hear what latencies you > >>>>> measure with the changes in place. > >>>>> > >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if > >>>>> you built cryptopp by default) to see if there is a difference. > >>>>> There is a ton of boilerplate setting up encryption contexts and > >>>>> key structures and so on that I suspect could be cached (perhaps > >>>>> stashed in the CryptoKey struct?) with a bit of effort. See > >>>>> > >>>>> > >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L2 > >>>>> 13 > >>>>> > >>>>> sage > >>>>> -- > >>>>> To unsubscribe from this list: send the line "unsubscribe > >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org > >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>>> > >>>>> > >>>> -- > >>>> To unsubscribe from this list: send the line "unsubscribe > >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>> -- To unsubscribe from this list: send the line "unsubscribe > >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> -- > >> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de > >> ITXperts GmbH http://www.itxperts.de > >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 > >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > >> > >> Company details: http://www.itxperts.de/imprint.htm > >> > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@vger.kernel.org More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2015-03-10 20:55 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-01-22 23:55 wip-auth Sage Weil 2015-01-26 6:20 ` wip-auth Blinick, Stephen L 2015-01-26 17:23 ` wip-auth Sage Weil 2015-01-26 18:00 ` wip-auth Blinick, Stephen L 2015-01-26 18:16 ` wip-auth Mark Nelson 2015-01-28 1:10 ` wip-auth Blinick, Stephen L 2015-01-27 17:18 ` wip-auth Sage Weil 2015-01-30 7:10 ` wip-auth Andreas Bluemle 2015-01-30 21:08 ` wip-auth Sage Weil 2015-02-04 23:24 ` wip-auth Mark Nelson [not found] ` <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv> 2015-02-06 7:45 ` wip-auth Alexandre DERUMIER 2015-02-09 18:24 ` wip-auth Mark Nelson [not found] ` <987318816.223626.1423542893540.JavaMail.zimbra@oxygem.tv> 2015-02-10 4:36 ` wip-auth Alexandre DERUMIER 2015-03-10 20:54 ` wip-auth Blinick, Stephen L 2015-03-10 20:55 ` wip-auth Sage Weil
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.