Bug: fetch with deepen shortens history

* Bug: fetch with deepen shortens history
@ 2023-06-26 21:24 Benjamin Stein
  2023-07-25  0:03 ` Benjamin Stein
  0 siblings, 1 reply; 4+ messages in thread
From: Benjamin Stein @ 2023-06-26 21:24 UTC (permalink / raw)
  To: git

Hello git gurus,

Here's an atypical bug report for you. I'm sorry for not starting with the template, but the context/setup are longer than felt useful in that format.

I have what I believe to be a (relatively) simple, reproducible test case (repo setup/steps below) around shallow checkouts at merge commits and deepening where the behavior is quite surprising - I end up with a smaller history after a fetch operation than when I started!

I've tried this on multiple OSes (Linux, Mac/Darwin), shells (zsh, bash) and various git versions ranging from 2.40.1 to 2.34.1, but I suspect it's older than that.

Scenario:
I'm using GitHub actions to look through some commit history and generate a report of commits relative to another branch. The specifics aren't super important, just that I start off in a shallow repo (depth=1) because (1) that's what GHA drops me into by default, and (2) since this is a large mono-repo I don't want to fetch all history every time - so I want to minimize the amount of data fetched.
So I used `fetch --shallow-exclude=other-branch HEAD` to get the relevant commits, and ran into my bug: when I do an extra `fetch --deepen` I end up with only a single commit in history instead of the N I had right before the call. If that's not super clear, I think the reproduction steps below should help.

Even more confusingly, if I check out a merge commit into trunk (the default) it misbehaves, but if I instead start out on the branch I'm inspecting, the same sequence of commands works correctly! This detail threw me for a long time as I tried to understand why behavior after a merge wasn't consistent.

My hunch is that this has to do with .git/shallow and the way some of the SHAs are coalescing when histories intersect/are combined, but I'm not too familiar with the inner workings of shallow. I even tried running the same sequence of steps with various combinations of --update-shallow on the fetches, but that doesn't seem to address the underlying issue of history shortening.

I hope that's all clear. If I can give more detail, definitely let me know, and I'm happy to try and explore some solutions, but I'm not certain where to begin.
I'm also not sure if it's an issue on the client or server - since I was initially testing with GitHub - but in my reduced example everything is local.

Thanks for your time and help with this!

-Benji

------ requisite bugreport answers ------

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)
See setup below.
In my shallow checkout, I ran:
    git log --oneline | wc -l => N commits of history.
Then I ran
    git fetch --deepen=1 <branch>
Followed by
    git log --oneline | wc -l => now just one commit.

What did you expect to happen? (Expected behavior)
I expected to see N+1 commits of history because I deepened by 1.

What happened instead? (Actual behavior)
I instead had just one commit of history.

What's different between what you expected and what actually happened?
I expected not to have my local history shortened by ~N commits despite using --deepen instead of --depth.

Anything else you want to add:
See additional notes/steps in this email and attached script to reproduce

[System Info]
git version:
git version 2.40.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 6.2.6-76060206-generic #202303130630~1681329778~22.04~d824cd4 SMP PREEMPT_DYNAMIC Wed A x86_64
compiler info: gnuc: 11.3
libc info: glibc: 2.35
$SHELL (typically, interactive shell): /bin/zsh

[Enabled Hooks]
not run from a git repository - no hooks to show
(it's irrelevant, my example starts from nothing, so no hooks)

------- bug-setup.sh ----- (better to to run in chunks, rather than all at once, but provided for convenience)

set -x
# Setup working folder for easy cleanup
mkdir git-test && cd git-test

# Setup sample repo
mkdir source-repo && cd source-repo
git init
git branch -m trunk
for i in {01..05}; do echo "start${i}" >> start; git add start; git commit -m "start${i}"; done
git branch old-checkpoint
for i in {01..10}; do echo "new${i}" >> new; git add new; git commit -m "new${i}"; done
git checkout -b feature HEAD~2
for i in {01..03}; do echo "feature${i}" >> feature; git add feature; git commit -m "feature${i}"; done
git checkout trunk
git merge --no-edit feature
cd ..
sleep 1

# Checkout shallow clone at feature branch - this works as desired
git clone --no-local source-repo --depth=1 --branch feature shallow-clone-feature
cd shallow-clone-feature
git remote set-branches --add origin '*'
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 12, also as expected
cd ..
sleep 1

# Checkout shallow clone at merge commit - this illustrates the bug
git clone --no-local source-repo --depth=1 --branch trunk shallow-clone-merge
cd shallow-clone-merge
git remote set-branches --add origin '*'
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 1, unexpected

# Wait, what? Let's try that again
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, still as expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 13, different this time. Unexpected.
cd ..
sleep 1

# What if we expand depth first?
git clone --no-local source-repo --depth=1 --branch trunk shallow-clone-with-depth
cd shallow-clone-with-depth
git remote set-branches --add origin '*'
git fetch --depth=2 origin feature
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 1, still unexpected
cd ..
sleep 1

# It turns out the depth query sometimes works if I also manually include HEAD, but still strangely.
# If I use depth=2, it's fine, but if I keep depth=3 it's not.
git clone --no-local source-repo --depth=1 --branch trunk shallow-clone-with-depth
cd shallow-clone-with-depth
git remote set-branches --add origin '*'
git fetch --depth=3 origin HEAD feature # it works if I use or include HEAD, but *not* if I include old-checkpoint
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 4, different from before, still wrong
cd ..
sleep 1

# If we start with deepen, instead
git clone --no-local ./source-repo --depth=1 --branch trunk shallow-clone-deepen
cd shallow-clone-deepen
git remote set-branches --add origin '*'
git fetch --deepen=1 origin HEAD feature # this also works if we use feature
git fetch origin --shallow-exclude=old-checkpoint feature
git log --oneline origin/feature | wc -l # 11, expected
git fetch --deepen=1 origin feature
git log --oneline origin/feature | wc -l # 12, that's finally correct
cd ..
sleep 1

# manually clear everything out by running
# cd .. && rm -rf git-test

^ permalink raw reply	[flat|nested] 4+ messages in thread