encrypted repositories?

encrypted repositories?

[Matthias Andree]

Greetings,

I have a rather special usage scenario.

Assume you have a repository where you want to work on embargoed  
information, so that not even system administrators of the server you're  
pushing to can get a hold of the cleartext data.

"Server" would be a central reference repository that I can push to.
"Client" would by my working computer that has a clone of the crypted  
repo, and an unencrypted checkout of it. Perhaps the client would also  
need an unencrypted copy of the repo (for performance reasons, I'm not  
sure about that) that gets encrypted on the fly when pushing and decrypted  
when fetching.

Examples of use might be press releases of upcoming products, written  
exams for students, whatever.

Requirements:
- "client" that is about to push must encrypt the data before pushing it  
to the server.
- all data (including file names, log messages,

Allowed restrictions:
- "server" limited to bare repositories
- initial version limited to symmetric encryption with pre-shared secret

In a later step, some key management and asymmetric crypto would be  
useful, but that's not crucial now. In my current scenario, those who are  
working on the embargoed material would trust one another.


How would one go about this from the user side? I sincerely doubt I have  
the resources (time!) to actually implement this in Git.

TIA

-- 
Matthias Andree

Re: encrypted repositories?

[Michael J Gruber]

Matthias Andree venit, vidit, dixit 17.07.2009 17:14:
> Greetings,
> 
> I have a rather special usage scenario.
> 
> Assume you have a repository where you want to work on embargoed  
> information, so that not even system administrators of the server you're  
> pushing to can get a hold of the cleartext data.
> 
> "Server" would be a central reference repository that I can push to.
> "Client" would by my working computer that has a clone of the crypted  
> repo, and an unencrypted checkout of it. Perhaps the client would also  
> need an unencrypted copy of the repo (for performance reasons, I'm not  
> sure about that) that gets encrypted on the fly when pushing and decrypted  
> when fetching.
> 
> Examples of use might be press releases of upcoming products, written  
> exams for students, whatever.
> 
> Requirements:
> - "client" that is about to push must encrypt the data before pushing it  
> to the server.
> - all data (including file names, log messages,
> 
> Allowed restrictions:
> - "server" limited to bare repositories
> - initial version limited to symmetric encryption with pre-shared secret
> 
> In a later step, some key management and asymmetric crypto would be  
> useful, but that's not crucial now. In my current scenario, those who are  
> working on the embargoed material would trust one another.
> 
> 
> How would one go about this from the user side? I sincerely doubt I have  
> the resources (time!) to actually implement this in Git.

If the server can not decrypt anything then it can not serve anything,
at least not as a git server. Note that if you're really fussy about
security then you should not allow the server to see even the DAG (which
would be the case if you encrypt blobs only), which makes it impossible
to do any smart serving.

So, why not share some form of remote storage on which you have an
encrypted luks partition? That way you can even set up multiple access
keys and revoke them when necessary.

Michael

Re: encrypted repositories?

[Matthias Kestenholz]

On Fri, Jul 17, 2009 at 3:14 PM, Matthias Andree<matthias.andree@gmx.de> wrote:
>
> How would one go about this from the user side? I sincerely doubt I have the
> resources (time!) to actually implement this in Git.
>

Maybe you could send around packages created by git-bundle as
pgp-encrypted emails, and keep the original on your computer on an
encrypted filesystem? You could also encrypt the bundles and put them
onto a public server if that suits you better than email.

The biggest advantage of this approach would be that it can be done
now, and no changes to git itself are required.


Matthias

Re: encrypted repositories?

[Linus Torvalds]

On Fri, 17 Jul 2009, Matthias Andree wrote:
>
> Assume you have a repository where you want to work on embargoed information,
> so that not even system administrators of the server you're pushing to can get
> a hold of the cleartext data.

If the server can't ever read it, you're basically limited to just one 
story:

 - use rsync-like "stupid" transports to upload and download things.

 - a "smart" git server (eg the native git:// style protocol is not going 
   to be possible)

and you strictly speaking need no real git changes, because you might as 
well just do it by uploading an encrypted tar-file of the .git directory. 
And there is literally no upside in doing anything else - any native git 
support is almost entirely pointless.

You could make it a _bit_ more useful perhaps by adding some helper 
wrappers, probably by just implementing a new transport name (ie instead 
of using "rsync://", you'd just use "crypt-tgz://" or something).

Now, that said, there are probably situations where maybe you'd allow the 
server to decrypt things _temporarily_, but you don't want to be encrypted 
on disk, and no persistent keys on the server, then that would open up a 
lot more possibilities.

Of course, that still does require that you trust the server admin to 
_some_ degree - anybody who has root would be able to get the keys by 
running a debugger on the git upload/download sequence when you do a 
upload or download.

Maybe that kind of security is still acceptable to you, though? 

IF that is the case, then at least in theory we could add support for 
"encryption key exchange" to the native git protocol, and then you could 
have encryption over the network access (ssh obviously already does that, 
but I'm including things like the anonymous git:// protocol too), and 
you'd have encrypted data on disk, but git-upload-pack would be able to 
decrypt things in order to do deltas etc.

But see above: in order for that to work, you do have to allow the pack 
upload and download processes on the server to decrypt things (in memory). 
So it would not be "absolutely secure".

			Linus

Re: encrypted repositories?

[Jakub Narebski]

Michael J Gruber <git@drmicha.warpmail.net> writes:
> Matthias Andree venit, vidit, dixit 17.07.2009 17:14:
> > 
> > I have a rather special usage scenario.
> > 
> > Assume you have a repository where you want to work on embargoed  
> > information, so that not even system administrators of the server you're  
> > pushing to can get a hold of the cleartext data.
> > 
> > "Server" would be a central reference repository that I can push to.
> > "Client" would by my working computer that has a clone of the crypted  
> > repo, and an unencrypted checkout of it. Perhaps the client would also  
> > need an unencrypted copy of the repo (for performance reasons, I'm not  
> > sure about that) that gets encrypted on the fly when pushing and decrypted  
> > when fetching.
> > 
> > Examples of use might be press releases of upcoming products, written  
> > exams for students, whatever.
 
> If the server can not decrypt anything then it can not serve anything,
> at least not as a git server. Note that if you're really fussy about
> security then you should not allow the server to see even the DAG (which
> would be the case if you encrypt blobs only), which makes it impossible
> to do any smart serving.

There was shown here on git mailing list script which was meant to
help in situation where you have repository with sensitive deta,
discovered repository corruption or bug in git, and cannot be
reproduced otherwise.  But I think it didn't encrypt repositry, but
just emulate it's structure.

As to encrypting repository: you can encrypt blobs (content of files),
you can encrypt filenames (but the structure remains) or you can put
files in a flat encrypted structure, and you can encrypt commit
messages and comitter and author info, and encrypt / rename branch
names.  Still some DAG structure will be visible, and need be visible
for "smart" git server (access via ssh and git protocols) to work.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: encrypted repositories?

[John Tapsell]

2009/7/17 Linus Torvalds <torvalds@linux-foundation.org>:
>
>
> On Fri, 17 Jul 2009, Matthias Andree wrote:
>>
>> Assume you have a repository where you want to work on embargoed information,
>> so that not even system administrators of the server you're pushing to can get
>> a hold of the cleartext data.
>
> If the server can't ever read it, you're basically limited to just one
> story:

Why couldn't you have the actual code encrypted, but have the server
still know about the SHAs etc?  You would expose the actual commit
structure, but that might be acceptable?

John

Re: encrypted repositories?

[Linus Torvalds]

On Fri, 17 Jul 2009, John Tapsell wrote:
> 
> Why couldn't you have the actual code encrypted, but have the server
> still know about the SHAs etc?  You would expose the actual commit
> structure, but that might be acceptable?

Even that wouldn't really work, because you'd never be able to generate 
any deltas.

So there would be no real advantage. In fact, there would be only 
disadvantages, because without any delta generation, you'd now have to 
actually transfer _more_ data.

			Linus

Re: encrypted repositories?

[Linus Torvalds]

On Fri, 17 Jul 2009, Linus Torvalds wrote:
> 
> On Fri, 17 Jul 2009, John Tapsell wrote:
> > 
> > Why couldn't you have the actual code encrypted, but have the server
> > still know about the SHAs etc?  You would expose the actual commit
> > structure, but that might be acceptable?
> 
> Even that wouldn't really work, because you'd never be able to generate 
> any deltas.
> 
> So there would be no real advantage. In fact, there would be only 
> disadvantages, because without any delta generation, you'd now have to 
> actually transfer _more_ data.

Oh, if you let the server know all the SHA's at _all_ levels (ie down to 
the blob itself), and then just make the blobs be encrypted, we'd be able 
to do some trivial optimizations, like only sending the actual blobs that 
changed. HOWEVER. That would reveal absolutely tons of data about the 
repository, and about the history. You'd have lost a _lot_ of security.

			Linus

Re: encrypted repositories? with git-torrent?

[Thomas Koch]

Wouldn't this be a use case for git-torrent?
http://code.google.com/p/gittorrent/
http://repo.or.cz/w/VCS-Git-Torrent.git

As I understand it, all data would be stored decentraliced and the (optional?) 
central server only saves, who has which objects.

(Just hearing a podcast on bittorrent while reading GIT mailinglist :-)

Thomas Koch, http://www.koch.ro

Re: encrypted repositories?

[Matthias Andree]

Am 17.07.2009, 21:38 Uhr, schrieb Linus Torvalds  
<torvalds@linux-foundation.org>:

>
>
> On Fri, 17 Jul 2009, Matthias Andree wrote:
>>
>> Assume you have a repository where you want to work on embargoed  
>> information,
>> so that not even system administrators of the server you're pushing to  
>> can get
>> a hold of the cleartext data.
>
> If the server can't ever read it, you're basically limited to just one
> story:
>
>  - use rsync-like "stupid" transports to upload and download things.
>
>  - a "smart" git server (eg the native git:// style protocol is not going
>    to be possible)

I don't know all its features, apparently it's online recompression - this  
is no longer going to be available.

> and you strictly speaking need no real git changes, because you might as
> well just do it by uploading an encrypted tar-file of the .git directory.
> And there is literally no upside in doing anything else - any native git
> support is almost entirely pointless.
>
> You could make it a _bit_ more useful perhaps by adding some helper
> wrappers, probably by just implementing a new transport name (ie instead
> of using "rsync://", you'd just use "crypt-tgz://" or something).
>
> Now, that said, there are probably situations where maybe you'd allow the
> server to decrypt things _temporarily_, but you don't want to be  
> encrypted
> on disk, and no persistent keys on the server, then that would open up a
> lot more possibilities.

> Of course, that still does require that you trust the server admin to
> _some_ degree - anybody who has root would be able to get the keys by
> running a debugger on the git upload/download sequence when you do a
> upload or download.
>
> Maybe that kind of security is still acceptable to you, though?

No, the server can't be allowed access to the keys or decrypted data.

I'm not sure about the graph, and if I should be concerned. Exposing the  
DAG might be in order.

It would be ok if the disk storage and the over-the-wire format cannot use  
delta compression then. It would suffice to just send a set of objects  
efficiently - and perhaps smaller revisions can be delta-compressed by the  
clients when pushing.

I admit haven't checked how the current git:// over-the-wire protocol[s]  
work[s]. I think client-side delta compression may require limiting the  
graph depths or delta size (when exceeded, the client must send the  
standalone self-contained object rather than a delta), so that the server  
can refuse patches when the delta nesting or size gets too deep/big.

I think this would generate the git server to something like a storage  
device for objects, perhaps with the DAG if exposed.


On a more general note, is someone looking into improving the http://  
efficiency?  Perhaps there are synergies between my plan of (a) encryption  
and (b) more efficient "dumb" (http/rsync/...) protocol use.

-- 
Matthias Andree

Re: encrypted repositories? with git-torrent?

[Matthias Andree]

Am 18.07.2009, 21:09 Uhr, schrieb Thomas Koch <thomas@koch.ro>:

> Wouldn't this be a use case for git-torrent?
> http://code.google.com/p/gittorrent/
> http://repo.or.cz/w/VCS-Git-Torrent.git
>
> As I understand it, all data would be stored decentraliced and the  
> (optional?) central server only saves, who has which objects.

I wonder about latency and accessibility here if clients are disconnected.  
Seems this is more for transferring repositories to large numbers of  
customers as sort of content distribution network for high load, rather  
than low connectivity - and the latter is my prime concern and also a  
detail of my scenario.

-- 
Matthias Andree

Re: encrypted repositories?

[Jakub Narebski]

"Matthias Andree" <matthias.andree@gmx.de> writes:

> On a more general note, is someone looking into improving the http://
> efficiency?  Perhaps there are synergies between my plan of (a)
> encryption  and (b) more efficient "dumb" (http/rsync/...) protocol
> use.

There was idea about improving http:// efficiency, but it was via
crating git-over-HTTP aka. "smart" HTTP server, i.e. you would have to
have DAG exposed, like for git:// and ssh://


On the other hand for http:// server need only "dumb" web server, and
additional metadata generated by git-update-server-info.  It is client
who does "walking" the DAG, so all data including server metadata can
be encrypted, and decrypted on-the-fly by client.  

I don't know though what information leakage you would get from
existence of loose objects and packfiles, and their sizes.  Probably
negligible...

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: encrypted repositories?

[Jeff King]

On Mon, Jul 20, 2009 at 02:09:28PM +0200, Matthias Andree wrote:

> No, the server can't be allowed access to the keys or decrypted data.
> 
> I'm not sure about the graph, and if I should be concerned. Exposing
> the DAG might be in order.
> 
> It would be ok if the disk storage and the over-the-wire format
> cannot use delta compression then. It would suffice to just send a
> set of objects efficiently - and perhaps smaller revisions can be
> delta-compressed by the clients when pushing.

The problem is that you need to expose not just the DAG, but also the
hashes of trees and blobs. Because if I know you have master^, and I want
to send you master, then I need to know which objects are referenced by
master that are not referenced by master^.

So now you have security implications, because I can do an offline
guessing attack against your files (i.e., calculate git blob hashes for
likely candidates and see if you have them). Whether that is a problem
really depends on your data.

Not to mention that it makes the protocol a lot more complex, as you
would be encrypting _parts_ of objects, like the filenames of a tree,
and the commit message of a commit object.

I suppose in theory you could obfuscate the sha1's in a way that
preserved the object relationships but revealed no information. That is,
the server would have one "fake" set of sha1's, and the client would map
its real sha1's to the fake ones when talking with the server. But that
is again potentially getting complex.

-Peff

Re: encrypted repositories?

[Matthias Andree]

Am 20.07.2009, 17:30 Uhr, schrieb Jeff King <peff@peff.net>:

> On Mon, Jul 20, 2009 at 02:09:28PM +0200, Matthias Andree wrote:
>
>> No, the server can't be allowed access to the keys or decrypted data.
>>
>> I'm not sure about the graph, and if I should be concerned. Exposing
>> the DAG might be in order.
>>
>> It would be ok if the disk storage and the over-the-wire format
>> cannot use delta compression then. It would suffice to just send a
>> set of objects efficiently - and perhaps smaller revisions can be
>> delta-compressed by the clients when pushing.
>
> The problem is that you need to expose not just the DAG, but also the
> hashes of trees and blobs. Because if I know you have master^, and I want
> to send you master, then I need to know which objects are referenced by
> master that are not referenced by master^.

Yes, you need to know that.  Not all of the push logic needs to be  
implemented on the server though.

In my scenario, the server degenerates into sort of a general object store  
- I really don't expect much smartness there. What is easily available  
(clients providing deltas rather than full objects) could be exploited,  
and that's it.

We can always have two local repositories, one reference and one checkout.  
The reference is a decrypted (unencrypted) copy of the set of objects on  
the server, and I could use that for tracking the server-side view (for  
instance, what are master^ and master pointing at so I can derive git  
rev-list master^...master, what do I need to send to the server).

I'm well aware that crypto requires more efforts on the client side if we  
don't trust the server, that's just natural.

The question is: which VCS can serve my scenario?

> So now you have security implications, because I can do an offline
> guessing attack against your files (i.e., calculate git blob hashes for
> likely candidates and see if you have them). Whether that is a problem
> really depends on your data.

Or look at commit frequency and push sources. There's always a leak of  
information even if I just upload a series of  
blah-2009MMDD-NNN.tar.lzma.gpg files... The data is going to be obsolete,  
say, 3 months; students then write the exam and then it's sort of public  
anyways. Even if your model does not entail not publishing exams (as  
opposed to embargoed press releases under development), but you can't  
prevent someone from writing their recollection of the problems from  
memory afterwards and sharing it with other students.

> Not to mention that it makes the protocol a lot more complex, as you
> would be encrypting _parts_ of objects, like the filenames of a tree,
> and the commit message of a commit object.
>
> I suppose in theory you could obfuscate the sha1's in a way that
> preserved the object relationships but revealed no information. That is,
> the server would have one "fake" set of sha1's, and the client would map
> its real sha1's to the fake ones when talking with the server. But that
> is again potentially getting complex.

Is your concern that the object name (SHA1) is derived from the  
unencrypted version?

-- 
Matthias Andree

Re: encrypted repositories?

[Matthias Andree]

Am 20.07.2009, 15:48 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:

> "Matthias Andree" <matthias.andree@gmx.de> writes:
>
>> On a more general note, is someone looking into improving the http://
>> efficiency?  Perhaps there are synergies between my plan of (a)
>> encryption  and (b) more efficient "dumb" (http/rsync/...) protocol
>> use.
>
> There was idea about improving http:// efficiency, but it was via
> crating git-over-HTTP aka. "smart" HTTP server, i.e. you would have to
> have DAG exposed, like for git:// and ssh://
>
>
> On the other hand for http:// server need only "dumb" web server, and
> additional metadata generated by git-update-server-info.  It is client
> who does "walking" the DAG, so all data including server metadata can
> be encrypted, and decrypted on-the-fly by client.

Fine by me, and seems to be some "minimal disclosure to server".

> I don't know though what information leakage you would get from
> existence of loose objects and packfiles, and their sizes.  Probably
> negligible...

Dunno. Given that it's just a collection of object sizes, you can't tell  
 from the SHA1 if the object in question is tree, tag, blob, or commit.

I'm really not after on-the-fly delta re-compression on the server-side  
for crypto stuff. I'm more thinking along the lines of zsync/bsdiff/xdelta  
(http://zsync.moria.org.uk/ for the least-known) - but zsync can't work on  
encrypted data. Perhaps encrypting the diffs could work, but then what's  
the difference to using the http:// and update-server-info related  
material and combining that with client-side on-the-fly (de/en)cryption?

-- 
Matthias Andree

Re: encrypted repositories?

[Jeff King]

On Tue, Jul 21, 2009 at 10:25:50AM +0200, Matthias Andree wrote:

> >The problem is that you need to expose not just the DAG, but also the
> >hashes of trees and blobs. Because if I know you have master^, and I want
> >to send you master, then I need to know which objects are referenced by
> >master that are not referenced by master^.
> 
> Yes, you need to know that.  Not all of the push logic needs to be
> implemented on the server though.

Yes, though fetching is much harder, since the server is the one holding
the information about how the transfer can be optimized. Still, you
should be able to achieve roughly the same performance as http fetching
from a dumb server.

> Or look at commit frequency and push sources. There's always a leak
> of information even if I just upload a series of
> blah-2009MMDD-NNN.tar.lzma.gpg files... The data is going to be
> obsolete, say, 3 months; students then write the exam and then it's
> sort of public anyways. Even if your model does not entail not
> publishing exams (as opposed to embargoed press releases under
> development), but you can't prevent someone from writing their
> recollection of the problems from memory afterwards and sharing it
> with other students.
> [...]
> Is your concern that the object name (SHA1) is derived from the
> unencrypted version?

Yes. You are potentially leaking considerable information about the
unencrypted contents which an attacker could use to guess those contents
(especially if the file is mostly composed of low-entropy parts, like
text formatting).

-Peff

Re: encrypted repositories?

[J-S-B]

To Matthias.

My name is John Brumbelow, and I know how to solve your issue. Its requires
client-driven-encryption, such that the server never gets the data
un-encrypted, ever, and thus can never decrypt it, but yet, on behalf of the
client, and other clients, users can still search, partially search,
wild-char-search, document-reference-search, and most important, sort data
stored on the server, that the server (and its administrators) can never
"see" cause it never left the client making/editing it, un-encrypted.

Note the key phrase is client-driven-encryption, not, client-encryption.
That is, the client drives the encryption process in conjunction with the
server(s).

This makes it so the data can never be stolen from the server, even if a
villan showed up at the server and held the administrators hostage, or at
any point beyond the client.

There is more...

I also know how to further obfuscate the data, as it is being made, to
protect the end user, so if the villan showed up at the client, and held
them hostage and/or hacked their computer(s), their data would still be
protected. This is not just a simple, extra-hash over the data, but
something that gives the villan fake/false data, which could be
traced/tracked, by the server/authorities.

Please email me at John-S-Brumbelow@hotmail.com





--
View this message in context: http://git.661346.n2.nabble.com/encrypted-repositories-tp3275970p7564308.html
Sent from the git mailing list archive at Nabble.com.