[PATCH 0/5] patchwork-bot: Use worktrees for repos

[PATCH 0/5] patchwork-bot: Use worktrees for repos

[Kees Cook]

Hi,

To use patchwork-bot locally, I wanted to use my existing worktrees for
various repos. This makes a few small improvements and then adds logic
to deal with specifying branchs and operating in worktrees.

Thanks!

-Kees

Kees Cook (5):
  patchwork-bot: Note which actions are part of "housekeeping"
  patchwork-bot: Create CACHEDIR if it is missing
  patchwork-bot: Allow limiting heads to single branch
  patchwork-bot: Handle repo being a worktree
  patchwork-bot: Allow initial database to skip recent commits

 git-patchwork-bot.example.yaml |  3 ++
 git-patchwork-bot.py           | 50 ++++++++++++++++++++++++++++------
 2 files changed, 45 insertions(+), 8 deletions(-)

-- 
2.30.2

[PATCH 1/5] patchwork-bot: Note which actions are part of "housekeeping"

[Kees Cook]

Expand the --help text slightly to remind users what things the
"housekeeping" option does.

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 git-patchwork-bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-patchwork-bot.py b/git-patchwork-bot.py
index f87af6ef41d6..d1eac6c0c0df 100755
--- a/git-patchwork-bot.py
+++ b/git-patchwork-bot.py
@@ -1325,7 +1325,7 @@ if __name__ == '__main__':
     parser.add_argument('-v', '--verbose', action='store_true', default=False,
                         help='Be more verbose in logging output')
     parser.add_argument('-k', '--housekeeping', action='store_true', default=False,
-                        help='Perform a housekeeping run')
+                        help='Perform a housekeeping run (supersede, archive)')
     parser.add_argument('--cachedir', default=None,
                         help='Cache directory to use instead of ~/.cache/git-patchwork-bot')
     parser.add_argument('--domain', default=None,
-- 
2.30.2

[PATCH 2/5] patchwork-bot: Create CACHEDIR if it is missing

[Kees Cook]

Create the CACHEDIR if it is missing (e.g. the first time
git-patchwork-bot is executed).

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 git-patchwork-bot.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/git-patchwork-bot.py b/git-patchwork-bot.py
index d1eac6c0c0df..dc10fd0effa9 100755
--- a/git-patchwork-bot.py
+++ b/git-patchwork-bot.py
@@ -1375,6 +1375,9 @@ if __name__ == '__main__':
         cfgyaml = fh.read()
     CONFIG = ruamel.yaml.safe_load(cfgyaml)
 
+    if not os.path.isdir(CACHEDIR):
+        os.makedirs(CACHEDIR, exist_ok=True)
+
     if cmdargs.housekeeping:
         for _pserver, _sconfig in CONFIG['patchworks'].items():
             for _pname in _sconfig['projects']:
-- 
2.30.2

[PATCH 3/5] patchwork-bot: Allow limiting heads to single branch

[Kees Cook]

To sanely use a single git repository with multiple worktrees, it must
be possible to limit the commits by branch (otherwise all commits are
visible for all worktrees). Add an optional "branch" argument for this
limit.

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 git-patchwork-bot.example.yaml | 2 ++
 git-patchwork-bot.py           | 6 +++---
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/git-patchwork-bot.example.yaml b/git-patchwork-bot.example.yaml
index d6140ed68e53..c3f6c1ea3133 100644
--- a/git-patchwork-bot.example.yaml
+++ b/git-patchwork-bot.example.yaml
@@ -52,6 +52,8 @@ repos:
   'pub/scm/linux/kernel/git/khilman/linux-amlogic.git':
     # For the notification emails
     treename: 'khilman/linux-amlogic.git'
+    # Optional: limit checking to a single branch (instead of --heads)
+    #branch: 'origin/master'
     # For the notification emails, must include %s
     commitlink: 'https://git.kernel.org/khilman/linux-amlogic/c/%.12s'
     # Only notify for commits where committer email exactly matches one
diff --git a/git-patchwork-bot.py b/git-patchwork-bot.py
index dc10fd0effa9..fc4d9e384122 100755
--- a/git-patchwork-bot.py
+++ b/git-patchwork-bot.py
@@ -409,9 +409,9 @@ def git_run_command(gitdir, args, stdin=None):
     return output
 
 
-def git_get_repo_heads(gitdir):
+def git_get_repo_heads(gitdir, branch):
     refs = list()
-    lines = git_get_command_lines(gitdir, ['show-ref', '--heads'])
+    lines = git_get_command_lines(gitdir, ['show-ref', branch])
     if lines is not None:
         for line in lines:
             (commit_id, refname) = line.split()
@@ -1049,7 +1049,7 @@ def housekeeping(pname):
 
 
 def pwrun(repo, rsettings):
-    git_heads = git_get_repo_heads(repo)
+    git_heads = git_get_repo_heads(repo, branch=rsettings.get('branch', '--heads'))
     if not git_heads:
         logger.info('Could not get the latest ref in %s', repo)
         sys.exit(1)
-- 
2.30.2

[PATCH 4/5] patchwork-bot: Handle repo being a worktree

[Kees Cook]

Locate the real git location when operating on a worktree, allowing
multiple pw.db files to be stored in a single top-level git repo.

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 git-patchwork-bot.example.yaml |  1 +
 git-patchwork-bot.py           | 27 +++++++++++++++++++++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/git-patchwork-bot.example.yaml b/git-patchwork-bot.example.yaml
index c3f6c1ea3133..94ab334f5043 100644
--- a/git-patchwork-bot.example.yaml
+++ b/git-patchwork-bot.example.yaml
@@ -53,6 +53,7 @@ repos:
     # For the notification emails
     treename: 'khilman/linux-amlogic.git'
     # Optional: limit checking to a single branch (instead of --heads)
+    # This becomes a required setting if the repo path is a worktree.
     #branch: 'origin/master'
     # For the notification emails, must include %s
     commitlink: 'https://git.kernel.org/khilman/linux-amlogic/c/%.12s'
diff --git a/git-patchwork-bot.py b/git-patchwork-bot.py
index fc4d9e384122..9be945d92a8a 100755
--- a/git-patchwork-bot.py
+++ b/git-patchwork-bot.py
@@ -1054,8 +1054,28 @@ def pwrun(repo, rsettings):
         logger.info('Could not get the latest ref in %s', repo)
         sys.exit(1)
 
+    dbpath = repo
+    # If we're aimed at a worktree, move up from the ".git" file to
+    # the worktree directory.
+    if not os.path.isdir(dbpath):
+        gitdir = open(dbpath).readline().strip()
+        if not gitdir.startswith('gitdir: '):
+            logger.info('Could not find git tree in %s', dbpath)
+            sys.exit(1)
+        gitdir = gitdir.split(' ', 1)[1]
+        gitdir, worktree = os.path.split(gitdir)
+        gitdir, category = os.path.split(gitdir)
+        if category != "worktrees":
+            logger.info('Could not find git worktree in %s', dbpath)
+            sys.exit(1)
+        # To store multiple pw.db files in a single .git directory,
+        # add a suffix based on the repo treename.
+        treename = rsettings.get('treename').replace('/', '_')
+        dbpath = os.path.join(gitdir, f'pw-{treename}.db')
+    else:
+        dbpath = os.path.join(dbpath, 'pw.db')
+
     # Do we have a pw.db there yet?
-    dbpath = os.path.join(repo, 'pw.db')
     db_exists = os.path.isfile(dbpath)
     dbconn = sqlite3.connect(dbpath, sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES)
     c = dbconn.cursor()
@@ -1295,10 +1315,13 @@ def check_repos():
 
     for repo in CONFIG['repos']:
         fullpath = os.path.join(cmdargs.reposdir.rstrip('/'), repo.lstrip('/'))
-        if not os.path.isdir(fullpath):
+        if not os.path.exists(fullpath):
             logger.info('Repository not found: %s', repo)
             continue
         settings = CONFIG['repos'][repo]
+        if not os.path.isdir(fullpath) and settings.get('branch', None) == None:
+            logger.info('Worktree must specify "branch" setting: %s', repo)
+            continue
         logger.info('Processing: %s', repo)
         pwrun(fullpath, settings)
 
-- 
2.30.2

[PATCH 5/5] patchwork-bot: Allow initial database to skip recent commits

[Kees Cook]

To test patchwork-bot, it is helpful to create the initial database
without a certain number of the latest commits, so that on the next
execution, the "missing" commits will appear new, and will be used for
processing state changes. Add the --ancestors argument (defaulting to
0, the existing behavior), to provide this ability.

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 git-patchwork-bot.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/git-patchwork-bot.py b/git-patchwork-bot.py
index 9be945d92a8a..2ebe5b4e88e4 100755
--- a/git-patchwork-bot.py
+++ b/git-patchwork-bot.py
@@ -409,13 +409,17 @@ def git_run_command(gitdir, args, stdin=None):
     return output
 
 
-def git_get_repo_heads(gitdir, branch):
+def git_get_repo_heads(gitdir, branch, ancestry=None):
     refs = list()
     lines = git_get_command_lines(gitdir, ['show-ref', branch])
+    if ancestry == None:
+        ancestry = ''
+    else:
+        ancestry = f'~{ancestry}'
     if lines is not None:
         for line in lines:
             (commit_id, refname) = line.split()
-            refs.append((refname, commit_id))
+            refs.append((refname, commit_id + ancestry))
     return refs
 
 
@@ -1082,7 +1086,9 @@ def pwrun(repo, rsettings):
 
     if not db_exists:
         db_init_pw_sqlite_db(c)
-        db_save_repo_heads(c, git_heads)
+        initial_git_heads = git_get_repo_heads(repo, branch=rsettings.get('branch', '--heads'),
+                                               ancestry=cmdargs.ancestors)
+        db_save_repo_heads(c, initial_git_heads)
         # Exit early
         dbconn.commit()
         return
@@ -1353,6 +1359,8 @@ if __name__ == '__main__':
                         help='Cache directory to use instead of ~/.cache/git-patchwork-bot')
     parser.add_argument('--domain', default=None,
                         help='Domain to use when creating message-ids')
+    parser.add_argument('--ancestors', default=None,
+                        help='During initial database creation, consider this many ancestor commits as fresh')
 
     cmdargs = parser.parse_args()
 
-- 
2.30.2

Re: [PATCH 0/5] patchwork-bot: Use worktrees for repos

[Konstantin Ryabitsev]

On Tue, 30 Nov 2021 16:11:21 -0800, Kees Cook wrote:
> To use patchwork-bot locally, I wanted to use my existing worktrees for
> various repos. This makes a few small improvements and then adds logic
> to deal with specifying branchs and operating in worktrees.
> 
> Thanks!
> 
> -Kees
> 
> [...]

Applied, thanks!

[1/5] patchwork-bot: Note which actions are part of "housekeeping"
      commit: bba5718890cbc3704528c028e104dde197d4c49c
[2/5] patchwork-bot: Create CACHEDIR if it is missing
      commit: e2dc648b93b77e45fde736ecbff8d0080fca41ff
[3/5] patchwork-bot: Allow limiting heads to single branch
      commit: 36971ee3ad62a56f40c3663a82b7475151df2a77
[4/5] patchwork-bot: Handle repo being a worktree
      commit: 460990bdca13805f7e9b5d6c8037437f655bcfac
[5/5] patchwork-bot: Allow initial database to skip recent commits
      commit: 6fc2e91a876b0a2771eedb0fe6106d7f47eb8aab

Best regards,
-- 
Konstantin Ryabitsev <konstantin@linuxfoundation.org>