Re: msgr2 protocol

[Sage Weil <sweil@redhat.com>]

On Sat, 11 Jun 2016, Marcus Watts wrote:
> If the client doesn't look at "features" before it sends stuff, it
> will not be able to be very smart about taking advantage of some
> future better method.  In fact, there isn't much advantage
> to the server sending anything early - it could just as easily
> wait until after it's seen the clients request.
> 
> Failing hard & retrying on a failed reconnect is going to be slower.
> On the bright side, at least it shouldn't happen often.

Yep.  Well, I think it is the client's (limited choice).  If it needs to 
know the server features, it needs to either wait for them, or make some 
optimistic choice and be prepared to pay the cost of a mistake.  We should 
give the client choice, though, if we can.

> If you're sending encryption (w/ different auth or keys) from several
> different streams, how are you planning to indicate which bits
> go with which scheme?, and which bits are you planning to encrypt
> and which not?  

This is what he stream ids are for, and why the outer portion of the frame 
is unencrypted.  See

	https://github.com/ceph/ceph/pull/9461/files#diff-83789b4be697d82eedbcbe330c44b436R68

 +  stream_id (le32)
 +  frame_len (le32)
 +  tag (TAG_* byte)
 +  payload
 +  [payload padding -- only present after stream auth phase]
 +  [signature -- only present after stream auth phase]

The tag and payload (and padding) would be encrypted or signed, but not 
the stream id and frame_len.

> Byte count limits.  Basically, you don't want collisions because
> of duplicated keys or data.  This depends on your crypto system,
> so, for instance, you should not encrypt with one key more than
> 	aes, cbc	about 2^68 bytes
> 	aes, ctr	exactly 2^128 bytes
> more generally, this depends on mode, blocksize, ...
> This applies across *all* uses of the key - and so you would
> generally want to use the session key directly as little as possible.
> (in particular, using the session key for ctr directly would be very very bad.)
> 
> If you've got multiple streams going already, you should be able
> to include a fairly simple rekey method with little effort.
> For instance, as part of the method, you could,
> 	up front as part of the method
> 		send a per-stream key encrypted under the shared secret.
> 	prepend to the first data sent in a payload
> 		byte limit, stream key #0 (encrypted under the per-stream key)
> 		then encrypt the next N bytes with stream key #0
> 	when the byte limit is reached, prepend to the
> 		next data sent in a payload
> 		byte limit, stream key #1 (encrypted under the per-stream key)
> 		then encrypt the next N bytes with stream key #1
> 	&etc.

Good idea.  If I understand correctly, it means that the session_key is 
only used to send the new/next random encryption key, and if we make the 
byte limit part of the initial protocol we get the rotation we need.  It 
might be simpler to do it as a frame limit instead of byte limit, and 
assume max-length frames (2^32 bytes).  We could still be super 
conservative and rotate the encryption key every 2^16 messages or 
something...?  And rotating the key on frame boundaries should be much 
simpler to implement.

Anyway, that part can be defined a bit later, I think.

Thanks!
sage