[PATCH v2 0/3] Additional FAQ entries

[PATCH v2 0/3] Additional FAQ entries

[brian m. carlson]

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

I've opted not to include backing up repositories in the syncing patch
because I think they're separate topics.  We could well end up with an
additional FAQ entry on that topic, which is left as an exercise for the
reader.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

brian m. carlson (3):
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees

 Documentation/gitfaq.txt | 90 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 84 insertions(+), 6 deletions(-)

[PATCH v2 1/3] gitfaq: add documentation on proxies

[brian m. carlson]

Many corporate environments and local systems have proxies in use.  Note
the situations in which proxies can be used and how to configure them.
At the same time, note what standards a proxy must follow to work with
Git.  Explicitly call out certain classes that are known to routinely
have problems reported various places online, including in the Git for
Windows issue tracker and on Stack Overflow, and recommend against the
use of such software.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 946691c153..5c21951f7b 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,29 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+[[proxy]]
+Can I use a proxy with Git?::
+	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
+	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
+	it also can be configured with `http.proxy` and similar options for HTTPS (see
+	linkgit:git-config[1]).  The `http.proxy` and related options can be
+	customized on a per-URL pattern basis.  In addition, Git can in theory
+	function normally with transparent proxies that exist on the network.
++
+For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
+include `netcat` and `socat`.  However, they must be configured not to exit when
+seeing EOF on standard input, which usually means that `netcat` will require
+`-q` and `socat` will require a timeout with something like `-t 10`.
++
+Note that in all cases, for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+
 Design
 ------
 

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

[Eric Sunshine]

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.

A couple minor comments; may not be worth a reroll or discussion...

> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> +[[proxy]]
> +Can I use a proxy with Git?::
> +       Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +       `https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +       it also can be configured with `http.proxy` and similar options for HTTPS (see
> +       linkgit:git-config[1]).  The `http.proxy` and related options can be
> +       customized on a per-URL pattern basis.  In addition, Git can in theory
> +       function normally with transparent proxies that exist on the network.
> ++
> +For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
> +include `netcat` and `socat`.  However, they must be configured not to exit when
> +seeing EOF on standard input, which usually means that `netcat` will require
> +`-q` and `socat` will require a timeout with something like `-t 10`.

I've seen this come up on the mailing list a couple times recently,
though I haven't really followed along and don't use Git through an
SSH proxy, thus I did have to go do some reading to understand what
this is talking about. Perhaps people searching out this FAQ entry
will already have sufficient context to understand what this is
saying, so maybe no additional context is needed here. However, I was
wondering if it might make sense for this to give a bit of reason
explaining _why_ these tools need to be configured to not exit
immediately upon EOF. As it stands now, this solution is a black box;
it will work but people won't understand why. Perhaps that doesn't
matter since most people consulting a FAQ like this probably just want
to get the thing working and don't care about the underlying details.
Then again, if the underlying reason is made more readily apparent,
maybe this knowledge can become more widespread.

> +Note that in all cases, for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that

"modify, tamper with, change" sounds like it came from the Department
of Redundancy Department. I like the sound of "tamper with" since the
image it conveys feels quite suitable here. Perhaps this could be
simplified to:

   The proxy cannot tamper with or buffer the...

> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

On 2021-11-07 at 23:27:24, Eric Sunshine wrote:
> I've seen this come up on the mailing list a couple times recently,
> though I haven't really followed along and don't use Git through an
> SSH proxy, thus I did have to go do some reading to understand what
> this is talking about. Perhaps people searching out this FAQ entry
> will already have sufficient context to understand what this is
> saying, so maybe no additional context is needed here. However, I was
> wondering if it might make sense for this to give a bit of reason
> explaining _why_ these tools need to be configured to not exit
> immediately upon EOF. As it stands now, this solution is a black box;
> it will work but people won't understand why. Perhaps that doesn't
> matter since most people consulting a FAQ like this probably just want
> to get the thing working and don't care about the underlying details.
> Then again, if the underlying reason is made more readily apparent,
> maybe this knowledge can become more widespread.

I'll try to see if I can stuff in a sentence there about why that's
necessary.  I think I understand it sufficiently well to summarize it.

> "modify, tamper with, change" sounds like it came from the Department
> of Redundancy Department. I like the sound of "tamper with" since the
> image it conveys feels quite suitable here. Perhaps this could be
> simplified to:
> 
>    The proxy cannot tamper with or buffer the...

I realize this sounds redundant, but I'm trying to avoid the situation
where people say, "I'm not _tampering_ with it, since I'm authorized to
do this by the company.  I'm just modifying it to remove this
inappropriate content/malware/data leak."  My goal here is to make it
crystal clear that if you do this, you'll break things, and provide
ammunition for people to go to their IT departments and say, "Look, your
proxy prevents me from doing my job.  The Git developers say so.  Fix
it."

I can drop one of "change" and "modify", though, since I think they're
synonyms.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> I realize this sounds redundant, but I'm trying to avoid the situation
> where people say, "I'm not _tampering_ with it, since I'm authorized to
> do this by the company.  I'm just modifying it to remove this
> inappropriate content/malware/data leak."

It might be a good strategy to educate users by leaving a deliberate
hole to let them quibble with such an excuse.  When it comes to
on-the-wire protocol, their management do not have authority to
grant them right to tamper with the data and that is not limited to
Git.

Re: [PATCH v2 1/3] gitfaq: add documentation on proxies

[Johannes Schindelin]

Hi brian,

On Sun, 7 Nov 2021, brian m. carlson wrote:

> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for

Not a big issue, but I think there is an "at" or "to" missing before
"various places online".

> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 946691c153..5c21951f7b 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,29 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>
> +[[proxy]]
> +Can I use a proxy with Git?::
> +	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +	it also can be configured with `http.proxy` and similar options for HTTPS (see
> +	linkgit:git-config[1]).  The `http.proxy` and related options can be
> +	customized on a per-URL pattern basis.  In addition, Git can in theory
> +	function normally with transparent proxies that exist on the network.
> ++
> +For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
> +include `netcat` and `socat`.

The first idea I had after reading this is: where are these examples
documented? Certainly not where I expected them, namely at
https://git-scm.com/docs/git-config#Documentation/git-config.txt-coregitProxy

And then I got puzzled. Why would `gitproxy` be used for the _SSH_
protocol? And I don't think it is used. I think it is only used for
connections using the `git://` protocol. I might very easily be wrong, of
course.

> However, they must be configured not to exit when +seeing EOF on
> standard input, which usually means that `netcat` will require +`-q` and
> `socat` will require a timeout with something like `-t 10`.

IMHO it would make sense to add a concrete example, or maybe even two
concrete examples, one for `netcat` and one for `socat`.

> ++
> +Note that in all cases, for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that
> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.
> +

This is good advice.

Ciao,
Dscho

[PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

[brian m. carlson]

In the FAQ, we tell people how to use the text attribute, but we fail to
explain what to do with the eol attribute.  As we ourselves have
noticed, most shell implementations do not care for carriage returns,
and as such, people will practically always want them to use LF endings.
Similar things can be said for batch files on Windows, except with CRLF
endings.

Since these are common things to have in a repository, let's help users
make a good decision by recommending that they use the gitattributes
file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which
originally referred to "the following entry", even though a new entry
has been inserted in between.  The cross-reference notation should
prevent this from occurring and provide a link in formats, such as HTML,
which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 5c21951f7b..ae1b526565 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -406,8 +406,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::
 +
 You can store the files in the repository with Unix line endings and convert
 them automatically to your platform's line endings.  To do that, set the
-configuration option `core.eol` to `native` and see the following entry for
-information about how to configure files as text or binary.
+configuration option `core.eol` to `native` and see
+<<recommended-storage-settings,the question on recommended storage settings>>
+for information about how to configure files as text or binary.
 +
 You can also control this behavior with the `core.whitespace` setting if you
 don't wish to remove the carriage returns from your line endings.
@@ -469,14 +470,25 @@ references, URLs, and hashes stored in the repository.
 +
 We also recommend setting a linkgit:gitattributes[5] file to explicitly mark
 which files are text and which are binary.  If you want Git to guess, you can
-set the attribute `text=auto`.  For example, the following might be appropriate
-in some projects:
+set the attribute `text=auto`.
++
+With text files, Git will generally ensure that LF endings are used in the
+repository, and will honor `core.autocrlf` and `core.eol` to decide what options
+to use when checking files out.  You can also override this by specifying a
+particular line ending such as `eol=lf` or `eol=crlf` if those files must always
+have that ending in the working tree (e.g., for functionality reasons).
++
+For example, the following might be appropriate in some projects:
 +
 ----
 # By default, guess.
 *	text=auto
 # Mark all C files as text.
 *.c	text
+# Ensure all shell files end up with LF endings and all batch files end up
+# with CRLF endings in the working tree and both end up with LF in the repo.
+*.sh text eol=lf
+*.bat text eol=crlf
 # Mark all JPEG files as binary.
 *.jpg	binary
 ----

Re: [PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

[Eric Sunshine]

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> In the FAQ, we tell people how to use the text attribute, but we fail to
> explain what to do with the eol attribute.  As we ourselves have
> noticed, most shell implementations do not care for carriage returns,
> and as such, people will practically always want them to use LF endings.
> Similar things can be said for batch files on Windows, except with CRLF
> endings.
>
> Since these are common things to have in a repository, let's help users
> make a good decision by recommending that they use the gitattributes
> file to correctly check out the endings.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -469,14 +470,25 @@ references, URLs, and hashes stored in the repository.
> +With text files, Git will generally ensure that LF endings are used in the
> +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> +to use when checking files out.  You can also override this by specifying a
> +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> +have that ending in the working tree (e.g., for functionality reasons).
> ++
> +For example, the following might be appropriate in some projects:
>  +
>  ----
>  # By default, guess.
>  *      text=auto
>  # Mark all C files as text.
>  *.c    text
> +# Ensure all shell files end up with LF endings and all batch files end up
> +# with CRLF endings in the working tree and both end up with LF in the repo.
> +*.sh text eol=lf
> +*.bat text eol=crlf
>  # Mark all JPEG files as binary.
>  *.jpg  binary
>  ----

I like the concrete explanation in the commit message of why `.sh` and
`.bat` files need to be configured specially, and was expecting that
quite useful information to be repeated here in the body. That does
seem exactly like what someone would come to the FAQ searching for,
and even though the above content hints at it, the hint may be just
vague enough to be overlooked. Consequently, I'm wondering if it ought
to be spelled out more explicitly here. In particular, if we give
examples of actual error messages people might encounter when
attempting to run a CRLF shell script or an LF batch file, that would
really give a FAQ searcher/reader something to latch onto when trying
to solve a problem. This could be done as a lead-in paragraph
immediately before the "For example, the following...".

Also, following v1 review, I think you had intended[1]: s/end up with/have/

[1]: https://lore.kernel.org/git/YW9wgbN%2Fb8NkVp4z@camp.crustytoothpaste.net/

Re: [PATCH v2 2/3] gitfaq: give advice on using eol attribute in gitattributes

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

On 2021-11-07 at 23:48:00, Eric Sunshine wrote:
> I like the concrete explanation in the commit message of why `.sh` and
> `.bat` files need to be configured specially, and was expecting that
> quite useful information to be repeated here in the body. That does
> seem exactly like what someone would come to the FAQ searching for,
> and even though the above content hints at it, the hint may be just
> vague enough to be overlooked. Consequently, I'm wondering if it ought
> to be spelled out more explicitly here. In particular, if we give
> examples of actual error messages people might encounter when
> attempting to run a CRLF shell script or an LF batch file, that would
> really give a FAQ searcher/reader something to latch onto when trying
> to solve a problem. This could be done as a lead-in paragraph
> immediately before the "For example, the following...".

I think that we could definitely add some text here to explain why one
would want to use this, providing an example.

> Also, following v1 review, I think you had intended[1]: s/end up with/have/

I will fix that in v3.  Thanks for the reminder.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

[PATCH v2 3/3] gitfaq: add entry about syncing working trees

[brian m. carlson]

Users very commonly want to sync their working tree with uncommitted
changes across machines, often to carry across in-progress work or
stashes.  Despite this not being a recommended approach, users want to
do it and are not dissuaded by suggestions not to, so let's recommend a
sensible technique.

The technique that many users are using is their preferred cloud syncing
service, which is a bad idea.  Users have reported problems where they
end up with duplicate files that won't go away (with names like "file.c
2"), broken references, oddly named references that have date stamps
appended to them, missing objects, and general corruption and data loss.
That's because almost all of these tools sync file by file, which is a
great technique if your project is a single word processing document or
spreadsheet, but is utterly abysmal for Git repositories because they
don't necessarily snapshot the entire repository correctly.  They also
tend to sync the files immediately instead of when the repository is
quiescent, so writing multiple files, as occurs during a commit or a gc,
can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that
the repository is quiescent, so let's suggest that and dissuade people
from using cloud syncing tools.  Let's tell people about common things
they should be aware of before doing this and that this is still
potentially risky.  Additionally, let's tell people that Git's security
model does not permit sharing working trees across users in case they
planned to do that.  While we'd still prefer users didn't try to do
this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 47 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index ae1b526565..e5dab89d6c 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,49 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  This is
+	potentially risky and in some cases can cause repository corruption or data
+	loss.
++
+Usually, doing so will cause `git status` to need to re-read every file in the
+working tree.  Additionally, Git's security model does not permit sharing a
+working tree across untrusted users, so it is only safe to sync a working tree
+if it will only be used by a single user across all machines.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+If you do this, the recommended approach is to use `rsync -a --delete-after`
+(ideally with an encrypted connection such as with `ssh`) on the root of
+repository.  You should ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc` and operations invoked by your editor).
++
+Be aware that even with these recommendations, syncing in this way has some risk
+since it bypasses Git's normal integrity checking for repositories, so having
+backups is advised.  You may also with to do a `git fsck` to verify the
+integrity of your data on the destination system after syncing.
+
 Common Issues
 -------------
 

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

[Eric Sunshine]

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> Users very commonly want to sync their working tree with uncommitted
> changes across machines, often to carry across in-progress work or
> stashes.  Despite this not being a recommended approach, users want to
> do it and are not dissuaded by suggestions not to, so let's recommend a
> sensible technique.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -185,6 +185,49 @@ Then, you can adjust your push URL to use `git@example_author` or
> +[[sync-working-tree]]
> +How do I sync a working tree across systems?::
> +       First, decide whether you want to do this at all.  Git usually works better
> +       when you push or pull your work using the typical `git push` and `git fetch`
> +       commands and isn't designed to share a working tree across systems.  This is
> +       potentially risky and in some cases can cause repository corruption or data
> +       loss.

The phrase "usually works better" makes this statement feel weak, thus
it may not convey the potential severity of the issue. I wonder if
rewording it something like this would make the statement more
forceful:

    Git works best when you `git push` and `git pull` your work
    between machines; it is not designed to share a working tree
    across systems. [...]

> +Usually, doing so will cause `git status` to need to re-read every file in the
> +working tree.  Additionally, Git's security model does not permit sharing a
> +working tree across untrusted users, so it is only safe to sync a working tree
> +if it will only be used by a single user across all machines.
> ++
> +It is important not to use a cloud syncing service to sync any portion of a Git
> +repository, since this can cause corruption, such as missing objects, changed
> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of
> +it being updated, since that is very likely to cause incomplete or partial
> +updates and therefore data loss.

Taking into consideration that people who are experiencing such
corruption will likely include the name of the syncing service in
their search query, would it make sense to mention some well-known
services here in order to make it more likely that people will
actually find this entry? Something like this, perhaps:

    It is important not to use a cloud syncing service (such as DropBox,
    FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
    repository...

> +Therefore, it's better to push your work to either the other system or a central
> +server using the normal push and pull mechanism.  However, this doesn't always
> +preserve important data, like stashes, so some people prefer to share a working
> +tree across systems.
> ++
> +If you do this, the recommended approach is to use `rsync -a --delete-after`
> +(ideally with an encrypted connection such as with `ssh`) on the root of
> +repository.  You should ensure several things when you do this:
> ++
> +* There are no additional worktrees enabled for your repository.

I don't fully understand this restriction. Can you explain it (at
least here in the email discussion)?

> +* You are not using a separate Git directory outside of your repository root.

Same question about this restriction.

> +* You are comfortable with the destination directory being an exact copy of the
> +       source directory, _deleting any data that is already there_.
> +* The repository is in a quiescent state for the duration of the transfer (that
> +       is, no operations of any sort are taking place on it, including background
> +       operations like `git gc` and operations invoked by your editor).
> ++
> +Be aware that even with these recommendations, syncing in this way has some risk
> +since it bypasses Git's normal integrity checking for repositories, so having
> +backups is advised.  You may also with to do a `git fsck` to verify the
> +integrity of your data on the destination system after syncing.

s/with/wish/

In fact, as with "usually" above, "wish" may be too weak. Perhaps say
instead that it is "recommended" that you use `git fsck` to verify the
integrity.

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> Taking into consideration that people who are experiencing such
> corruption will likely include the name of the syncing service in
> their search query, would it make sense to mention some well-known
> services here in order to make it more likely that people will
> actually find this entry? Something like this, perhaps:
>
>     It is important not to use a cloud syncing service (such as DropBox,
>     FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
>     repository...

I do agree that in a repository being actively modified, any
backup/sync solution that works per-file fashion would not work
well.  But is "cloud" a good word to characterise and group these
per-file backup/sync solution?  

Doesn't rsync work the same per-file fashion, and the only reason
why it is a better fit is because it is not continuous, not
attempting to "sync" while the repository is in use, until the user
explicitly says "OK, I am ready to go home, so let's stop working
here and send everything over to home with rsync"?

>> +* There are no additional worktrees enabled for your repository.
>
> I don't fully understand this restriction. Can you explain it (at
> least here in the email discussion)?
>
>> +* You are not using a separate Git directory outside of your repository root.
>
> Same question about this restriction.

As long as you know what you are doing and catch everything in
quiescent state, you should be fine, I would think.

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Doesn't rsync work the same per-file fashion, and the only reason
> why it is a better fit is because it is not continuous, not
> attempting to "sync" while the repository is in use, until the user
> explicitly says "OK, I am ready to go home, so let's stop working
> here and send everything over to home with rsync"?

OK, so not "per-file" but "continuous" is the root problem, and
"cloud" would be a good word because all the popular ones share that
"continuous" trait.

This part of the proposed patch text may need rethinking a bit.

> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of

That is, "file by file" is not a problem per-se, "don't understand
the structure" is a huge problem, and "continuous" may contribute to
the problem further.

I wonder if you let the "cloud" services to continuously sync your
repository, then go quiescent for a while and then start touching
the destination, it would be sufficient, though.  The refs with
funny "2" suffixes and the like are the symptom of two sides
simultanously mucking with the same file (e.g ".git/refs/main") and
the "cloud sync" could not decide what the right final outcome is,
right?

I also wonder if we add a way to transfer reflog entries, that imply
the object reachability, say "git push --with-reflog", over the
wire, it would be sufficient to do everything with Git.

Before you go home, you'd do

    git stash save --untracked && git stash apply
    git push --mirror --with-reflog --with-stash

to save away modified and untracked files to a stash entry [*], and push
all the refs with their reflog entries (including refs/stash which
normally gets refused because it has only two levels).

    Side note. If there were a variant of "git stash save" that only
    saves away without modifying the working tree and the index, I'd
    use that single command instead of "save and immediately restore
    by applying" kludge.

Then at the destination, you'd figure out what the current branch
was (the stash message should record tha name of the branch), check
that branch out, and running "git stash pop" will give you pretty
much the same environment.

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3077 bytes --]

On 2021-11-09 at 00:10:05, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Doesn't rsync work the same per-file fashion, and the only reason
> > why it is a better fit is because it is not continuous, not
> > attempting to "sync" while the repository is in use, until the user
> > explicitly says "OK, I am ready to go home, so let's stop working
> > here and send everything over to home with rsync"?
> 
> OK, so not "per-file" but "continuous" is the root problem, and
> "cloud" would be a good word because all the popular ones share that
> "continuous" trait.
> 
> This part of the proposed patch text may need rethinking a bit.
> 
> > +or added files, broken refs, and a wide variety of other corruption.  These
> > +services tend to sync file by file and don't understand the structure of a Git
> > +repository.  This is especially bad if they sync the repository in the middle of
> 
> That is, "file by file" is not a problem per-se, "don't understand
> the structure" is a huge problem, and "continuous" may contribute to
> the problem further.

"File by file" is not a problem per se, but it is a problem if you don't
snapshot the repository or it's in use.  For example, if you used an LVM
snapshot on an in-use repository and then synced that, we'd guarantee
that the repository was fully consistent.  Practically nobody does that,
though.

But to clarify: the goal of this type of software is to sync single,
independent documents.  As such, these tools don't necessarily consider
that it's important to sync other files in the same directory at the
same time.  You can end up with cases where some parts of your
repository get synced, then some other files in other directories, then
other parts of the repository, resulting in different states over
different times.  There's just no guarantee about how they behave with
all of the files in a given repository because they aren't considered a
relevant unit.  That's why I mention that they sync file by file.  If
they did syncing directory by directory, we'd likely have a lot fewer
problems.

> I wonder if you let the "cloud" services to continuously sync your
> repository, then go quiescent for a while and then start touching
> the destination, it would be sufficient, though.  The refs with
> funny "2" suffixes and the like are the symptom of two sides
> simultanously mucking with the same file (e.g ".git/refs/main") and
> the "cloud sync" could not decide what the right final outcome is,
> right?

I think this is very risky.  Yes, usually the duplicate files are caused
by competing changes on both sides, but having seen users with this
problem before, I'm not sure if we can safely guarantee this is the only
case.

> I also wonder if we add a way to transfer reflog entries, that imply
> the object reachability, say "git push --with-reflog", over the
> wire, it would be sufficient to do everything with Git.

Yes, please.  I'd love to see this.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH v2 3/3] gitfaq: add entry about syncing working trees

[Eric Sunshine]

On Mon, Nov 8, 2021 at 5:34 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On 2021-11-08 at 00:12:14, Eric Sunshine wrote:
> > Taking into consideration that people who are experiencing such
> > corruption will likely include the name of the syncing service in
> > their search query, would it make sense to mention some well-known
> > services here in order to make it more likely that people will
> > actually find this entry? Something like this, perhaps:
> >
> >     It is important not to use a cloud syncing service (such as DropBox,
> >     FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
> >     repository...
>
> There are a lot of these services.  My preference is to not name
> specific ones here, just like we don't name specific ones when we state
> that you shouldn't use an antivirus or firewall other than the Windows
> default, mostly because I'm not interested in angering corporate
> lawyers.  My advice on this topic is always general: XYZ is a cloud
> syncing service, and you should not use any cloud syncing services for Git.

My "would it make sense" question arose from taking into consideration
how people are likely to use a search engine for a particular problem:

    BuzzingBee syncing corrupted git repository

Without naming specific services or tools, it seems much less likely
that people will find this FAQ entry, thus will end up posting to
StackOverflow (or wherever) anyhow -- despite your intention and
effort behind the FAQ in the first place. I'm thinking about
discoverability -- which is the same reason I suggested in my review
of the other patch that it might be a good idea to add actual error
messages a person might encounter with a CRLF shell script or an LF
batch file.

> Additionally, the popular options differ by region.  For example, there
> are some services which are probably popular in China which are not
> popular elsewhere, and I don't think it's a good idea to try to guess
> which ones happen to be most popular around the world.

The other way to look at it is that listing many popular services
(wherever they happen to be) makes it more likely that search engines
will lead them to this FAQ entry and alleviate the need to post a
question somewhere.

Anyhow, it was just a thought. I think I've said all I have to say on
this subject. As I mentioned in response to the cover letter, all of
my comments were of the bikeshedding variety, and I didn't see any
show-stoppers in the series as posted.

> > > +* There are no additional worktrees enabled for your repository.
> >
> > I don't fully understand this restriction. Can you explain it (at
> > least here in the email discussion)?
>
> If you sync the main repository and working tree, but not the worktrees
> themselves, then you end up with incorrect data in the .git directory,
> which contains information about the worktree.

I still don't follow. What incorrect data will be in the .git
directory? Do you mean the absolute path pointing from the .git
directory to the worktree?

> More importantly, it can
> contain absolute path information, which would be undesirable even if
> you did sync the worktrees, say, if you used two different usernames
> (and hence two different home directories) on the two systems.

Okay, I figured that that was probably one of your concerns, but it
was difficult to tell from the raw "no additional worktrees". On the
other hand, I can easily see people syncing between machines in which
they have the same username and same directory structure, so this
bullet point seems overly restrictive.

> > > +* You are not using a separate Git directory outside of your repository root.
> >
> > Same question about this restriction.
>
> Again, if you sync the root of the working tree but don't sync the
> separate Git directory, you won't have copied the index or any of the
> other data.

Okay, again, this is indeed what I thought you might have in mind, but
it was difficult to be sure. And, again, this seems overly restrictive
since there may be plenty of scenarios in which this works well.

I suppose both of these points would feel more reasonable if they
didn't sound like hard requirements, perhaps by explaining that the
simple case of no worktrees and no separate repository has the least
restrictions and is easiest to get up and running; more complex cases
can work too with the caveat that worktree and separate repository
directories ought to be synced too, and that pathnames need to be
identical on all machines.

Re: [PATCH v2 0/3] Additional FAQ entries

[Eric Sunshine]

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> This series introduces some additional Git FAQ entries on various
> topics.  They are all things I've seen in my professional life or on
> Stack Overflow, so I've written documentation.
>
> Changes from v1:
> * Drop the monorepo patch for now; I want to revise it further.
> * Reorder the working tree patch to place more warnings up front.
> * Mention core.gitproxy and socat.
> * Rephrase text in the EOL entry to read correctly and be easier to
>   understand.
> * Improve the commit message for the working tree FAQ entry to make it
>   clearer that users wish to transfer uncommitted changes.

Thanks for re-rolling. FAQ entries are, of course, bikeshedding
fodder, so I left a number of bikeshedding comments and pointed out
one minor typo. Whether any of the comments warrant a re-roll or are
even worth discussing is a matter of opinion, and they could certainly
all be addressed later (by someone) atop the current series if
desired; I don't think any of the comments were blockers.

Re: [PATCH v2 0/3] Additional FAQ entries

[Johannes Schindelin]

Hi Eric,

On Sun, 7 Nov 2021, Eric Sunshine wrote:

> On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > This series introduces some additional Git FAQ entries on various
> > topics.  They are all things I've seen in my professional life or on
> > Stack Overflow, so I've written documentation.
> >
> > Changes from v1:
> > * Drop the monorepo patch for now; I want to revise it further.
> > * Reorder the working tree patch to place more warnings up front.
> > * Mention core.gitproxy and socat.
> > * Rephrase text in the EOL entry to read correctly and be easier to
> >   understand.
> > * Improve the commit message for the working tree FAQ entry to make it
> >   clearer that users wish to transfer uncommitted changes.
>
> Thanks for re-rolling. FAQ entries are, of course, bikeshedding
> fodder,

That is one possible perspective. Another possible perspective is that FAQ
entries are the first thing users in distress might turn to, so the more
helpful they are, the better. Maybe it would make sense to focus on that
a bit more. As in your suggestion to name cloud sync providers by name, to
make the FAQ entry eminently more "findable". To me, that did not at all
feel like pointless bike-shedding, but more like a really valuable
improvement that users will appreciate (even if they will be oblivious of
the fact that they appreciate it, of course).

Ciao,
Dscho

Re: [PATCH v2 0/3] Additional FAQ entries

[Johannes Schindelin]

Hi brian,

On Sun, 7 Nov 2021, brian m. carlson wrote:

> This series introduces some additional Git FAQ entries on various
> topics.  They are all things I've seen in my professional life or on
> Stack Overflow, so I've written documentation.
>
> I've opted not to include backing up repositories in the syncing patch
> because I think they're separate topics.  We could well end up with an
> additional FAQ entry on that topic, which is left as an exercise for the
> reader.

Makes sense.

You hinted in the thread that you were planning on submitting a v3. I hope
that my feedback regarding `core.gitproxy` is still in time for that.

Thank you,
Dscho

Re: [PATCH v2 0/3] Additional FAQ entries

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

On 2022-01-04 at 13:54:52, Johannes Schindelin wrote:
> Hi brian,
> 
> On Sun, 7 Nov 2021, brian m. carlson wrote:
> 
> > This series introduces some additional Git FAQ entries on various
> > topics.  They are all things I've seen in my professional life or on
> > Stack Overflow, so I've written documentation.
> >
> > I've opted not to include backing up repositories in the syncing patch
> > because I think they're separate topics.  We could well end up with an
> > additional FAQ entry on that topic, which is left as an exercise for the
> > reader.
> 
> Makes sense.
> 
> You hinted in the thread that you were planning on submitting a v3. I hope
> that my feedback regarding `core.gitproxy` is still in time for that.

I can definitely include that in v3.  I'm hoping I can get to it later
this week or during the weekend before my schedule becomes busy again.

I appreciate the thoughtful comments.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]