From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sage Weil Subject: Re: msgr2 protocol Date: Sun, 12 Jun 2016 19:59:20 -0400 (EDT) Message-ID: References: <20160610190510.GA18999@degu.eng.arb.redhat.com> <20160611230503.GA18268@degu.eng.arb.redhat.com> Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Return-path: Received: from mx1.redhat.com ([209.132.183.28]:38852 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753389AbcFLX6z (ORCPT ); Sun, 12 Jun 2016 19:58:55 -0400 In-Reply-To: <20160611230503.GA18268@degu.eng.arb.redhat.com> Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Marcus Watts Cc: Gregory Farnum , Haomai Wang , ceph-devel On Sat, 11 Jun 2016, Marcus Watts wrote: > If the client doesn't look at "features" before it sends stuff, it > will not be able to be very smart about taking advantage of some > future better method. In fact, there isn't much advantage > to the server sending anything early - it could just as easily > wait until after it's seen the clients request. > > Failing hard & retrying on a failed reconnect is going to be slower. > On the bright side, at least it shouldn't happen often. Yep. Well, I think it is the client's (limited choice). If it needs to know the server features, it needs to either wait for them, or make some optimistic choice and be prepared to pay the cost of a mistake. We should give the client choice, though, if we can. > If you're sending encryption (w/ different auth or keys) from several > different streams, how are you planning to indicate which bits > go with which scheme?, and which bits are you planning to encrypt > and which not? This is what he stream ids are for, and why the outer portion of the frame is unencrypted. See https://github.com/ceph/ceph/pull/9461/files#diff-83789b4be697d82eedbcbe330c44b436R68 + stream_id (le32) + frame_len (le32) + tag (TAG_* byte) + payload + [payload padding -- only present after stream auth phase] + [signature -- only present after stream auth phase] The tag and payload (and padding) would be encrypted or signed, but not the stream id and frame_len. > Byte count limits. Basically, you don't want collisions because > of duplicated keys or data. This depends on your crypto system, > so, for instance, you should not encrypt with one key more than > aes, cbc about 2^68 bytes > aes, ctr exactly 2^128 bytes > more generally, this depends on mode, blocksize, ... > This applies across *all* uses of the key - and so you would > generally want to use the session key directly as little as possible. > (in particular, using the session key for ctr directly would be very very bad.) > > If you've got multiple streams going already, you should be able > to include a fairly simple rekey method with little effort. > For instance, as part of the method, you could, > up front as part of the method > send a per-stream key encrypted under the shared secret. > prepend to the first data sent in a payload > byte limit, stream key #0 (encrypted under the per-stream key) > then encrypt the next N bytes with stream key #0 > when the byte limit is reached, prepend to the > next data sent in a payload > byte limit, stream key #1 (encrypted under the per-stream key) > then encrypt the next N bytes with stream key #1 > &etc. Good idea. If I understand correctly, it means that the session_key is only used to send the new/next random encryption key, and if we make the byte limit part of the initial protocol we get the rotation we need. It might be simpler to do it as a frame limit instead of byte limit, and assume max-length frames (2^32 bytes). We could still be super conservative and rotate the encryption key every 2^16 messages or something...? And rotating the key on frame boundaries should be much simpler to implement. Anyway, that part can be defined a bit later, I think. Thanks! sage