Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Fri, 19 May 2023, Konstantin Ryabitsev wrote:

> Our mailing list infrastructure is uniquely tailored to public-inbox users,
> with messages being written to the archives *before* they are even sent out to
> subscribers, which helps speed up access for users who aren't using the

There is a certain advantage there regarding the problem we currently have 
where messages sent as HTML are missing from public-inbox archives, if the 
process that strips HTML parts takes place before feeding to public-inbox 
(whereas at present it takes place after that, I think).

(I consider it important that we do *not* insist on contributors meeting a 
shibboleth of sending plain-text email before they can interact with our 
mailing lists, especially for user lists such as libc-help - we should be 
friendly to people sending email in the ordinary form they are used to 
using today even if we might rather they didn't send HTML email - we 
shouldn't impose that view of proper email on them.)

> The major improvement of this vs. bugzilla's native email interface is that
> bugs can be created from mailing list discussions, not just the other way
> around. The bugbot will need to be slightly adjusted for CTI needs, but
> otherwise I think it would be a better solution than Bugzilla's native email
> integration. For example, when bugmail is sent to the mailing list and someone

Note that people can send email to Bugzilla in reply to a message they 
received directly from Bugzilla - not just in reply to a message on 
glibc-bugs.  I think it's important that both cases of replying should 
work automatically, however that's implemented.

(If someone replies to a glibc-bugs message and forgets to remove 
glibc-bugs from their reply, the list gets two copies of the message at 
present - one that the user sent directly, one forwarded by Bugzilla.  
Avoiding that duplication somehow would be nice to have, but entirely 
optional.)

Note also that Bugzilla is one of the hardest migration issues (for 
non-GCC projects), because (as I mentioned in one of the meetings) 
Sourceware Bugzilla is used by many different projects, most of which are 
not part of CTI.

> * Sending per-commit email to mailing lists -- this seems like a vestigial
>   feature from the pre-git past. Does it really serve any purpose? I am
>   fighting to kill a similar feature used on the kernel.org side, because I do
>   not see any remaining legitimate use for it -- it just creates a lot of mail
>   traffic that nobody really reviews.

I think it's extremely useful to have those messages for watching out for 
commits that shouldn't have gone in or went in in an unintended form (this 
is for human-determined "shouldn't" or "unintended" - not anything 
expected to be covered by automated checks).  And also to have a message 
ready-made to reply to when something was committed without being posted 
to the mailing list.

> * We do not currently allow direct hooks on the server side, because running
>   arbitrary scripts with the permissions of the git server is a very bad
>   anti-pattern when it comes to repo security. It also tends to make pushes
>   super slow and frustrating to developers, especially those working on slow
>   or unstable connections that can go rapidly stale, resulting in failed
>   pushes.
> 
>   So, we will need to wrap our collective heads around what your hooks are
>   doing now and implement them via alternative means. For example:

Certainly some kind of system for proxying / containing checks to limit 
what they can do if buggy makes sense - they don't need to run with the 
permissions of the git server, they can run in some kind of isolation, as 
long as they prevent bad commits from getting into the history.

>   * denying force-pushes to specific branches (rebase/non-fast-forward
>     updates) is a native feature of Gitolite

What about denying merge commits on master and release branches?

Note that GCC has its own custom set of namespaces for branches (under 
refs/users/ and refs/vendors/) and rules about what branch names may be 
created at all etc.  (Unfortunately there seem to be some missing checks 
for the case of lightweight tag creation, resulting in some improperly 
named tags being present.)

>   The only thing that would require more work is style checkers and other CI
>   functionality that really should not be happening on-push. I suggest that
>   this moves entirely to patchwork CI, discussed below.

I think those things *should* be happening on push - to prevent obviously 
bad history getting onto master or release branches (at least) in the 
first place.  For glibc, that's (apart from the non-fast-forward checks 
and rejecting merge commits on branches meant to have linear history):

* Disallowing lines with trailing whitespace in certain files.

* Disallowing commits with a mailing list address as the author email 
(even if your mailing list setup is designed to be DKIM-safe so author 
emails don't need rewriting, people might still use "git am" with old 
messages, so this remains relevant).

* Disallowing commit subject lines that look like a ChangeLog header.

* Disallowing single-word (or empty) commit subject lines.

For GCC there are several extra checks that I enumerated - including, in 
particular, the fairly complicated checks for ChangeLog format to avoid 
the nightly cron job updating ChangeLogs falling over.  (The check to 
avoid From-SVN: lines is similarly to prevent a commit that would cause 
subsequent problems for automation that relies on those lines only being 
present in commits converted from SVN in order to look up such a commit by 
SVN commit number.)

> ## Release tarballs
> 
> There are multiple ways of getting this done, including completely automated.
> For example, stable kernel releases are generated server-side by using and
> verifying the PGP signature found in a git note attached to the release tag.
> This doesn't work if the tarball is not directly generated from a git
> repository (for example, if config scripts must run first).

Is "git archive" output or similar actually stable enough that the server 
can reliably generate (at any time in the future, with a future git 
version) a tarball matching the tarball signature the release manager 
generated at release time, or is that not what you meant?

I'd certainly expect the tarballs served as release tarballs to be 
byte-for-byte identical to the ones at https://ftp.gnu.org/gnu/glibc/ (and 
the signatures likewise to be identical).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joel Brobecker]

> > * Sending per-commit email to mailing lists -- this seems like a vestigial
> >   feature from the pre-git past. Does it really serve any purpose? I am
> >   fighting to kill a similar feature used on the kernel.org side, because I do
> >   not see any remaining legitimate use for it -- it just creates a lot of mail
> >   traffic that nobody really reviews.
> 
> I think it's extremely useful to have those messages for watching out for 
> commits that shouldn't have gone in or went in in an unintended form (this 
> is for human-determined "shouldn't" or "unintended" - not anything 
> expected to be covered by automated checks).  And also to have a message 
> ready-made to reply to when something was committed without being posted 
> to the mailing list.

I strongly agree as well.

-- 
Joel

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Fri, May 19, 2023 at 10:47:10PM +0000, Joseph Myers wrote:
> > Our mailing list infrastructure is uniquely tailored to public-inbox users,
> > with messages being written to the archives *before* they are even sent out to
> > subscribers, which helps speed up access for users who aren't using the
> 
> There is a certain advantage there regarding the problem we currently have 
> where messages sent as HTML are missing from public-inbox archives, if the 
> process that strips HTML parts takes place before feeding to public-inbox 
> (whereas at present it takes place after that, I think).

Hm... no, there is no "stripping of HTML parts" (that would invalidate DKIM).
It is certainly possible to configure a mailing list so that HTML messages are
pre-moderated.

> (I consider it important that we do *not* insist on contributors meeting a 
> shibboleth of sending plain-text email before they can interact with our 
> mailing lists, especially for user lists such as libc-help - we should be 
> friendly to people sending email in the ordinary form they are used to 
> using today even if we might rather they didn't send HTML email - we 
> shouldn't impose that view of proper email on them.)

It's possible, but it is an empirical fact that permitting HTML messages
raises the amount of spam that goes through by several orders of magnitude.
Allowing it per-list is okay and normal (e.g. the list aimed at newbies and
people seeking occasional help). However, I think it is perfectly normal to
disallow it on development lists where one is already expected to follow a
fairly rigid set of rules.

> > * Sending per-commit email to mailing lists -- this seems like a vestigial
> >   feature from the pre-git past. Does it really serve any purpose? I am
> >   fighting to kill a similar feature used on the kernel.org side, because I do
> >   not see any remaining legitimate use for it -- it just creates a lot of mail
> >   traffic that nobody really reviews.
> 
> I think it's extremely useful to have those messages for watching out for 
> commits that shouldn't have gone in or went in in an unintended form (this 
> is for human-determined "shouldn't" or "unintended" - not anything 
> expected to be covered by automated checks).  And also to have a message 
> ready-made to reply to when something was committed without being posted 
> to the mailing list.

If you insist, though I still think this is the case of "everyone assumes that
someone else is paying attention to it." I am highly critical of setups that
fire messages into the ether in hopes that someone checks them. 

There are better ways of doing it that don't generate email, e.g.:

- compare commits against patchwork and alert when there is no match
- write all pushes to an audit log and have someone monitor it
  (e.g. see https://tlog.linderud.dev/)
- monitor the RSS feed of all commits (e.g.
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/atom/?h=master)

> > * We do not currently allow direct hooks on the server side, because running
> >   arbitrary scripts with the permissions of the git server is a very bad
> >   anti-pattern when it comes to repo security. It also tends to make pushes
> >   super slow and frustrating to developers, especially those working on slow
> >   or unstable connections that can go rapidly stale, resulting in failed
> >   pushes.
> > 
> >   So, we will need to wrap our collective heads around what your hooks are
> >   doing now and implement them via alternative means. For example:
> 
> Certainly some kind of system for proxying / containing checks to limit 
> what they can do if buggy makes sense - they don't need to run with the 
> permissions of the git server, they can run in some kind of isolation, as 
> long as they prevent bad commits from getting into the history.

That would take even longer to execute, and be more fragile, no? I usually get
lots of push-back from developers when their pushes don't return within a
second or two, so having a push take many minutes to complete seems like bad
user experience.

> >   * denying force-pushes to specific branches (rebase/non-fast-forward
> >     updates) is a native feature of Gitolite
> 
> What about denying merge commits on master and release branches?

I'm not sure I understand -- you mean, all commits must have a single parent
and have a fully linear structure? Doesn't that make it super difficult to
resolve conflicts?

Sorry, I'm coming from the opposite world where merge commits are encouraged
-- the cover letter of the series goes into the merge commit, and therefore
carries changelogs and links to all previous revisions of the series that got
applied.

> Note that GCC has its own custom set of namespaces for branches (under 
> refs/users/ and refs/vendors/) and rules about what branch names may be 
> created at all etc.  (Unfortunately there seem to be some missing checks 
> for the case of lightweight tag creation, resulting in some improperly 
> named tags being present.)

As long as it works with git, that's not a problem -- but I believe a lot of
visualization tools (like cgit) won't display anything that's not under
refs/heads.

> I think those things *should* be happening on push - to prevent obviously 
> bad history getting onto master or release branches (at least) in the 
> first place.  For glibc, that's (apart from the non-fast-forward checks 
> and rejecting merge commits on branches meant to have linear history):
> 
> * Disallowing lines with trailing whitespace in certain files.
> 
> * Disallowing commits with a mailing list address as the author email 
> (even if your mailing list setup is designed to be DKIM-safe so author 
> emails don't need rewriting, people might still use "git am" with old 
> messages, so this remains relevant).
> 
> * Disallowing commit subject lines that look like a ChangeLog header.
> 
> * Disallowing single-word (or empty) commit subject lines.

In my view these all should be happening on the client as pre-commit checks,
not on the server. Any reason why they can't run on the client, e.g. via a
framework like https://pre-commit.com/ ?

> For GCC there are several extra checks that I enumerated - including, in 
> particular, the fairly complicated checks for ChangeLog format to avoid 
> the nightly cron job updating ChangeLogs falling over.  (The check to 
> avoid From-SVN: lines is similarly to prevent a commit that would cause 
> subsequent problems for automation that relies on those lines only being 
> present in commits converted from SVN in order to look up such a commit by 
> SVN commit number.)

Also something that, in my view, should be running pre-commit on the client.

> > ## Release tarballs
> > 
> > There are multiple ways of getting this done, including completely automated.
> > For example, stable kernel releases are generated server-side by using and
> > verifying the PGP signature found in a git note attached to the release tag.
> > This doesn't work if the tarball is not directly generated from a git
> > repository (for example, if config scripts must run first).
> 
> Is "git archive" output or similar actually stable enough that the server 
> can reliably generate (at any time in the future, with a future git 
> version) a tarball matching the tarball signature the release manager 
> generated at release time, or is that not what you meant?

It's stable enough to not have changed in a decade, but we also only need
compatibility within a short period of time. We generate the tarball from the
git-note signature once, shortly after it is pushed. We don't need to be able
to generate the same tarball years later. Once the tarball is generated, it is
put into the primary storage and kept there.

-K

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Tue, 23 May 2023, Konstantin Ryabitsev wrote:

> > (I consider it important that we do *not* insist on contributors meeting a 
> > shibboleth of sending plain-text email before they can interact with our 
> > mailing lists, especially for user lists such as libc-help - we should be 
> > friendly to people sending email in the ordinary form they are used to 
> > using today even if we might rather they didn't send HTML email - we 
> > shouldn't impose that view of proper email on them.)
> 
> It's possible, but it is an empirical fact that permitting HTML messages
> raises the amount of spam that goes through by several orders of magnitude.
> Allowing it per-list is okay and normal (e.g. the list aimed at newbies and
> people seeking occasional help). However, I think it is perfectly normal to
> disallow it on development lists where one is already expected to follow a
> fairly rigid set of rules.

The toolchain lists universally allow mail from non-subscribers, and I 
think most can expect sometimes to be included in discussions involving 
people who are not particularly active developers of that particular 
project (but may still be relevant to include in that individual 
discussion).

When allowing HTML through to the lists (with or without any form of 
moderation) we should make sure public-inbox properly archives those 
messages that include HTML.

> If you insist, though I still think this is the case of "everyone assumes that
> someone else is paying attention to it." I am highly critical of setups that
> fire messages into the ether in hopes that someone checks them. 

People do in practice monitor the commit emails.

> There are better ways of doing it that don't generate email, e.g.:

For projects using email for development, commit emails have the obvious 
advantage of not adding yet another different system to watch.

> > What about denying merge commits on master and release branches?
> 
> I'm not sure I understand -- you mean, all commits must have a single parent
> and have a fully linear structure?

Yes.  We use linear history (on master and release branches) for GCC and 
glibc.  (Shared development branches use merge commits for merging master 
into the branch.  Ones in the users and vendors namespaces might use 
merges or might be rebased, depending on the policy for the individual 
branch.)

> Doesn't that make it super difficult to resolve conflicts?

No.

> > I think those things *should* be happening on push - to prevent obviously 
> > bad history getting onto master or release branches (at least) in the 
> > first place.  For glibc, that's (apart from the non-fast-forward checks 
> > and rejecting merge commits on branches meant to have linear history):
> > 
> > * Disallowing lines with trailing whitespace in certain files.
> > 
> > * Disallowing commits with a mailing list address as the author email 
> > (even if your mailing list setup is designed to be DKIM-safe so author 
> > emails don't need rewriting, people might still use "git am" with old 
> > messages, so this remains relevant).
> > 
> > * Disallowing commit subject lines that look like a ChangeLog header.
> > 
> > * Disallowing single-word (or empty) commit subject lines.
> 
> In my view these all should be happening on the client as pre-commit checks,
> not on the server. Any reason why they can't run on the client, e.g. via a
> framework like https://pre-commit.com/ ?

A server can't ensure that the user's clone has any particular custom 
configuration.  We have some optional configuration 
(gcc-git-customization.sh), but there's no requirement to use it.  
Checking on push is much more reliable than expecting custom configuration 
on the user's side.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Tue, May 23, 2023 at 07:34:52PM +0000, Joseph Myers wrote:
> > It's possible, but it is an empirical fact that permitting HTML messages
> > raises the amount of spam that goes through by several orders of magnitude.
> > Allowing it per-list is okay and normal (e.g. the list aimed at newbies and
> > people seeking occasional help). However, I think it is perfectly normal to
> > disallow it on development lists where one is already expected to follow a
> > fairly rigid set of rules.
> 
> The toolchain lists universally allow mail from non-subscribers, and I 
> think most can expect sometimes to be included in discussions involving 
> people who are not particularly active developers of that particular 
> project (but may still be relevant to include in that individual 
> discussion).
> 
> When allowing HTML through to the lists (with or without any form of 
> moderation) we should make sure public-inbox properly archives those 
> messages that include HTML.

Yes, that's always been the case -- public-inbox does not do any filtering, it
just archives whatever the mailing list allows through.

> > If you insist, though I still think this is the case of "everyone assumes that
> > someone else is paying attention to it." I am highly critical of setups that
> > fire messages into the ether in hopes that someone checks them. 
> 
> People do in practice monitor the commit emails.

Okay, if you insist -- after all, we do the same thing for all kernel commits.
I just find it a pretty terrible waste of resources, as I'm pretty sure 99% of
everyone who subscribes to that list has set things up to go to a folder that
they never check.

> > > I think those things *should* be happening on push - to prevent obviously 
> > > bad history getting onto master or release branches (at least) in the 
> > > first place.  For glibc, that's (apart from the non-fast-forward checks 
> > > and rejecting merge commits on branches meant to have linear history):
> > > 
> > > * Disallowing lines with trailing whitespace in certain files.
> > > 
> > > * Disallowing commits with a mailing list address as the author email 
> > > (even if your mailing list setup is designed to be DKIM-safe so author 
> > > emails don't need rewriting, people might still use "git am" with old 
> > > messages, so this remains relevant).
> > > 
> > > * Disallowing commit subject lines that look like a ChangeLog header.
> > > 
> > > * Disallowing single-word (or empty) commit subject lines.
> > 
> > In my view these all should be happening on the client as pre-commit checks,
> > not on the server. Any reason why they can't run on the client, e.g. via a
> > framework like https://pre-commit.com/ ?
> 
> A server can't ensure that the user's clone has any particular custom 
> configuration.

Okay, perhaps I'm not fully grokking the picture. How many people are able to
push to master/release branches? I was expecting it to be 1-2, maybe a handful
of people, but perhaps it's a much wider set of developers?

-K

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Tue, 23 May 2023, Konstantin Ryabitsev wrote:

> Okay, perhaps I'm not fully grokking the picture. How many people are able to
> push to master/release branches? I was expecting it to be 1-2, maybe a handful
> of people, but perhaps it's a much wider set of developers?

There are 612 users in the gcc group.  There are 121 users in the glibc 
group.  There are 461 users in the src group (which can commit to the 
binutils-gdb repository).  (Obviously there's a lot of overlap between the 
users in those groups - people who have commit access to more than one 
project.)  There are no technical restrictions on which users in the right 
group can push to which branch; there are of course policies on patch 
approval before pushing, but mostly they are not enforced by technical 
means.

(It would be reasonable to e.g. only set up commit access on the new 
system automatically for users who have pushed to a repository within the 
past year or two - something that could be identified most reliably from 
the archives of the commit mailing lists, converting back from the 
committer information in each commit to a Sourceware username would be 
more complicated - and wait for others to request restoration of commit 
access.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Tue, May 23, 2023 at 08:12:10PM +0000, Joseph Myers wrote:
> On Tue, 23 May 2023, Konstantin Ryabitsev wrote:
> 
> > Okay, perhaps I'm not fully grokking the picture. How many people are able to
> > push to master/release branches? I was expecting it to be 1-2, maybe a handful
> > of people, but perhaps it's a much wider set of developers?
> 
> There are 612 users in the gcc group.  There are 121 users in the glibc 
> group.  There are 461 users in the src group (which can commit to the 
> binutils-gdb repository).

Okay, clearly a totally different approach to the kernel world where only a
single person (or perhaps 2-3 people) can write to a repository like
torvalds/linux.git or stable/linux.git. Hence my misunderstanding of what is
happening.

Can you please describe the lifecycle of a patch and how it gets from "user
sends a change to the mailing list" to "that patch becomes a commit in the
official project repository where it lives forever."

E.g. for the kernel, it is:

- the patch receives initial review by the subsystem maintainer
- after a few iterations, the patch is committed to that maintainer's
  separate repository (either on kernel.org or elsewhere)
- during the merge window, the maintainer sends a pull request to the mainline
  maintainer (Linus) who does a final review and either rejects it or merges
  it into the mainline

I need to more clearly understand the kind of access 612 users of the gcc
group have to the git repository.

-K

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Tue, 23 May 2023, Konstantin Ryabitsev wrote:

> Can you please describe the lifecycle of a patch and how it gets from "user
> sends a change to the mailing list" to "that patch becomes a commit in the
> official project repository where it lives forever."

A patch is sent to the list.  There are some number of review / revision 
rounds, ending with a reviewer or maintainer saying it is OK to commit (or 
OK after some specified changes, or OK in the absence of objections within 
some given period, etc.).  If the person posting the patch has commit 
access, they commit it; otherwise they should ask the maintainer to do so 
for them.  Maintainers of parts of the compiler may commit their patches 
without someone else reviewing them.  (There are also reviewers who can 
approve other people's patches in their area of the compiler but not their 
own - and global reviewers can approve patches anywhere in the compiler.)  
Once someone has had a few good patches approved and committed by someone 
else, they'll probably be asked to complete the form for getting commit 
access, naming the relevant maintainer as approver (only maintainers can 
approve commit access, non-maintainers with commit access can't do so).

Anyone with commit access can create their own branches in their 
refs/users/ namespace if they wish, and use them without needing review 
for commits going there.  Shared development branches under 
refs/heads/devel/ have whatever rules are established by the people 
setting up those branches.

Rules about patch approval are not enforced automatically, only the basic 
checks for certain kinds of problems with commits and for such things as 
non-fast-forward pushes only being allowed in namespaces such as 
refs/users/ and refs/vendors/.

The above is for GCC, but the general idea is similar for other projects, 
with differences of detail in the ref namespaces used, the checks carried 
out on commits and exactly how maintainership works.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-23 15:52, Konstantin Ryabitsev wrote:
>>> If you insist, though I still think this is the case of "everyone assumes that
>>> someone else is paying attention to it." I am highly critical of setups that
>>> fire messages into the ether in hopes that someone checks them.
>>
>> People do in practice monitor the commit emails.
> 
> Okay, if you insist -- after all, we do the same thing for all kernel commits.
> I just find it a pretty terrible waste of resources, as I'm pretty sure 99% of
> everyone who subscribes to that list has set things up to go to a folder that
> they never check.
> 

There's probably a viable middle ground here for glibc, where commits 
that don't have an entry in patchwork get sent to the list.  Given the 
current traffic for libc-alpha, it won't add much overhead to libc-alpha 
and also meet auditing requirements.

For the release/* branches, it probably makes more sense to send the 
commits to libc-stable, thus removing the need for developers to send 
[committed X.Y] to the list.  This should then limit the utility of 
glibc-cvs to private branch pushes.  I doubt if anybody cares about 
pushes to private branches, do they?

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/22/23 06:11, Joel Brobecker wrote:
>>> * Sending per-commit email to mailing lists -- this seems like a vestigial
>>>   feature from the pre-git past. Does it really serve any purpose? I am
>>>   fighting to kill a similar feature used on the kernel.org side, because I do
>>>   not see any remaining legitimate use for it -- it just creates a lot of mail
>>>   traffic that nobody really reviews.
>>
>> I think it's extremely useful to have those messages for watching out for 
>> commits that shouldn't have gone in or went in in an unintended form (this 
>> is for human-determined "shouldn't" or "unintended" - not anything 
>> expected to be covered by automated checks).  And also to have a message 
>> ready-made to reply to when something was committed without being posted 
>> to the mailing list.
> 
> I strongly agree as well.
> 

Likewise, I do post-commit review using the per-commit emails.

Even if we had an advanced process to review and merge code that process still needs
a belt-and-suspenders review from senior maintainers, and I do that by reviewing and
often responding to the per-commit emails to loop in the people that committed the
code.

The question for me is "What is lower cost?"

(a) Walking the git commit history, making a note of where I stopped and started,
    and then writing net new emails when I see something that needs correctly.
(b) Reviewing an mbox, deleting as required, and reply-all when I see something
    that needs review.

I like the per-commit emails because it lets me have some post-commit discussions
easily e.g. (b).

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/23/23 15:34, Joseph Myers wrote:
> A server can't ensure that the user's clone has any particular custom 
> configuration.  We have some optional configuration 
> (gcc-git-customization.sh), but there's no requirement to use it.  
> Checking on push is much more reliable than expecting custom configuration 
> on the user's side.
 
Is this a case where we are looking to solve a community enforcement issue with a technical
solution?

If we removed all the hooks, what is the consequence of a commit that is wrong? Can we just
fix the sources with a subsequent commit? Add a light-weight linting check in 'make check'
that additionally reports this?

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-24 08:44, Carlos O'Donell wrote:
> Likewise, I do post-commit review using the per-commit emails.
> 
> Even if we had an advanced process to review and merge code that process still needs
> a belt-and-suspenders review from senior maintainers, and I do that by reviewing and
> often responding to the per-commit emails to loop in the people that committed the
> code.
> 
> The question for me is "What is lower cost?"
> 
> (a) Walking the git commit history, making a note of where I stopped and started,
>      and then writing net new emails when I see something that needs correctly.
> (b) Reviewing an mbox, deleting as required, and reply-all when I see something
>      that needs review.
> 
> I like the per-commit emails because it lets me have some post-commit discussions
> easily e.g. (b).

Couldn't you use patchwork for this, filtering on Committed state?  If 
we have 100% coverage (sending non-matching commits to the ML will 
achieve that), you would be able to look through committed patches and 
then start the conversation in the patch thread instead of a separate 
one through glibc-cvs.

If noting where you stopped/started is cumbersome then you could add 
another Audited terminal state in patchwork to track that.

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 09:43, Siddhesh Poyarekar wrote:
> On 2023-05-24 08:44, Carlos O'Donell wrote:
>> Likewise, I do post-commit review using the per-commit emails.
>> 
>> Even if we had an advanced process to review and merge code that
>> process still needs a belt-and-suspenders review from senior
>> maintainers, and I do that by reviewing and often responding to the
>> per-commit emails to loop in the people that committed the code.
>> 
>> The question for me is "What is lower cost?"
>> 
>> (a) Walking the git commit history, making a note of where I
>> stopped and started, and then writing net new emails when I see
>> something that needs correctly. (b) Reviewing an mbox, deleting as
>> required, and reply-all when I see something that needs review.
>> 
>> I like the per-commit emails because it lets me have some
>> post-commit discussions easily e.g. (b).
> 
> Couldn't you use patchwork for this, filtering on Committed state?
> If we have 100% coverage (sending non-matching commits to the ML will
> achieve that), you would be able to look through committed patches
> and then start the conversation in the patch thread instead of a
> separate one through glibc-cvs.

Is 100% coverage an achievable and timely goal? :-)

I think that Patchwork *could* meet this need but isn't quite there yet.

> If noting where you stopped/started is cumbersome then you could add
> another Audited terminal state in patchwork to track that.

That would need to be a per-reviewer state that doesn't exist today in patchwork.

The Audited terminal state would be global, and Joseph and I notice different things
in our reviews and apply different criteria for calling out a patch that needs follow
up work or review.

In summary:

- Post-commit review facilitation for projects continues to be a value-add step
  in the maintainer process and is currently mbox-based.

- The bar for post-commit review today is an mbox review-style process and a
  replacement needs to handle multiple reviewers walking the queue at different
  rates and applying different criteria to review (tag-based scheme?)

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-24 10:12, Carlos O'Donell wrote:
> Is 100% coverage an achievable and timely goal? :-)

It's an achievable, timely *and* trivial goal if we post missing commits 
(i.e. commits in git without a matching patch on libc-alpha) to 
libc-alpha as [pushed].

> I think that Patchwork *could* meet this need but isn't quite there yet.
> 
>> If noting where you stopped/started is cumbersome then you could add
>> another Audited terminal state in patchwork to track that.
> 
> That would need to be a per-reviewer state that doesn't exist today in patchwork.
> 
> The Audited terminal state would be global, and Joseph and I notice different things
> in our reviews and apply different criteria for calling out a patch that needs follow
> up work or review.
> 
> In summary:
> 
> - Post-commit review facilitation for projects continues to be a value-add step
>    in the maintainer process and is currently mbox-based.
> 
> - The bar for post-commit review today is an mbox review-style process and a
>    replacement needs to handle multiple reviewers walking the queue at different
>    rates and applying different criteria to review (tag-based scheme?)

Wouldn't a [pushed] email to libc-alpha provide more visibility then, 
given that multiple people (I do too, albeit infrequently) tend to 
review these commits?  It shouldn't need a separate mailing list, given 
our email traffic volume.  Basically:

- Commits to the main branch go to libc-alpha
- Commits to release/* branches go to libc-stable
- All other commits (i.e. to private branches) get ignored.

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Tue, May 23, 2023 at 10:12:20PM +0000, Joseph Myers wrote:
> > Can you please describe the lifecycle of a patch and how it gets from "user
> > sends a change to the mailing list" to "that patch becomes a commit in the
> > official project repository where it lives forever."
> 
> A patch is sent to the list.  There are some number of review / revision 
> rounds, ending with a reviewer or maintainer saying it is OK to commit (or 
> OK after some specified changes, or OK in the absence of objections within 
> some given period, etc.).  If the person posting the patch has commit 
> access, they commit it; otherwise they should ask the maintainer to do so 
> for them.  Maintainers of parts of the compiler may commit their patches 
> without someone else reviewing them.

This workflow has one important problem -- you must ultimately trust the
integrity of your infrastructure. It would be very easy for a malicious actor
with access to backend infrastructure to insert a commit into the repository
without it being discovered:

1. they can override any server-side validations
2. they can disable email notifications
3. they can forge the Committer identity of the commit

With your current workflow nobody would find out that such commit has been
inserted into the history because everyone must rebase their commits anyway,
so having a push rejected due to newer remote commits is a routine situation
(as opposed to a being a huge red flag in a repository where only one or two
committers are allowed to write).

Similarly, since most commits end up being rebased, there can be no post-fact
validation to find out if there are any commits that did not go through
review -- their patch-id would likely no longer match the mailing list
submission anyway (at least, there would be too many false-positives).

This situation could be partially mitigated if commits were cryptographically
signed, but they don't appear to be.

GCC and other projects are sufficiently high targets that we should not trust
the infrastructure to be secure or admins to be above being bribed or forced
under duress.

> Anyone with commit access can create their own branches in their 
> refs/users/ namespace if they wish, and use them without needing review 
> for commits going there.  Shared development branches under 
> refs/heads/devel/ have whatever rules are established by the people 
> setting up those branches.

From the git backend perspective, this is not optimal, because the resulting
packs contain extra objects that aren't interesting to most cloners. For
example, if I only leave refs/heads and refs/tags in the gcc repository, I get
rid of over 1.1 million extra objects (from 3.9 million to 2.8 million) and
the pack shrinks from 1.8GB to 1.1GB.

To me, it makes more sense to separate this into multiple repositories and
relegate vendor/user branches to their own trees. They still share objects
behind the scenes, but packs and cloning is much more optimized.

-K

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-24 12:29, Konstantin Ryabitsev wrote:
> On Tue, May 23, 2023 at 10:12:20PM +0000, Joseph Myers wrote:
>>> Can you please describe the lifecycle of a patch and how it gets from "user
>>> sends a change to the mailing list" to "that patch becomes a commit in the
>>> official project repository where it lives forever."
>>
>> A patch is sent to the list.  There are some number of review / revision
>> rounds, ending with a reviewer or maintainer saying it is OK to commit (or
>> OK after some specified changes, or OK in the absence of objections within
>> some given period, etc.).  If the person posting the patch has commit
>> access, they commit it; otherwise they should ask the maintainer to do so
>> for them.  Maintainers of parts of the compiler may commit their patches
>> without someone else reviewing them.
> 
> This workflow has one important problem -- you must ultimately trust the
> integrity of your infrastructure. It would be very easy for a malicious actor
> with access to backend infrastructure to insert a commit into the repository
> without it being discovered:
> 
> 1. they can override any server-side validations
> 2. they can disable email notifications
> 3. they can forge the Committer identity of the commit
> 
> With your current workflow nobody would find out that such commit has been
> inserted into the history because everyone must rebase their commits anyway,
> so having a push rejected due to newer remote commits is a routine situation
> (as opposed to a being a huge red flag in a repository where only one or two
> committers are allowed to write).
> 
> Similarly, since most commits end up being rebased, there can be no post-fact
> validation to find out if there are any commits that did not go through
> review -- their patch-id would likely no longer match the mailing list
> submission anyway (at least, there would be too many false-positives).
> 
> This situation could be partially mitigated if commits were cryptographically
> signed, but they don't appear to be.
> 
> GCC and other projects are sufficiently high targets that we should not trust
> the infrastructure to be secure or admins to be above being bribed or forced
> under duress.

It looks like the question of who to ultimately trust, either the 
gatekeeper committer (or two) who is the only person to have write 
access to the repository, or the admin who manages the box.  The 
gatekeeper committer could also fabricate commits and push to the 
central repository in the same way.  In fact, the gatekeeper committer 
could choose to do worse, like delaying (or declining to merge) someones 
patches.

Signed commits would be nice, but we're not there yet as a community.

>  From the git backend perspective, this is not optimal, because the resulting
> packs contain extra objects that aren't interesting to most cloners. For
> example, if I only leave refs/heads and refs/tags in the gcc repository, I get
> rid of over 1.1 million extra objects (from 3.9 million to 2.8 million) and
> the pack shrinks from 1.8GB to 1.1GB.
> 
> To me, it makes more sense to separate this into multiple repositories and
> relegate vendor/user branches to their own trees. They still share objects
> behind the scenes, but packs and cloning is much more optimized.

That would be a nice optimization.  In fact, an interesting workflow 
hack here could be to have separate per-maintainer repositories that a 
central repository pulls from, subject to some checks passing.  It won't 
solve any of the trust problems since the repositories will all be in 
the same infrastructure, but it will add more validation in the 
pipeline, while reducing the main repository size.

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Wed, 24 May 2023, Siddhesh Poyarekar wrote:

> There's probably a viable middle ground here for glibc, where commits that
> don't have an entry in patchwork get sent to the list.  Given the current
> traffic for libc-alpha, it won't add much overhead to libc-alpha and also meet
> auditing requirements.

I think this is overly complicating things.  (Though it would be good for 
glibc to have the check done for GCC that once a commit is in the 
repository, emails aren't sent out again for a merge of that commit into a 
user / vendor / development branch.)

> For the release/* branches, it probably makes more sense to send the commits
> to libc-stable, thus removing the need for developers to send [committed X.Y]
> to the list.  This should then limit the utility of glibc-cvs to private
> branch pushes.  I doubt if anybody cares about pushes to private branches, do
> they?

I think commits to user / vendor branches can be of interest.  They may 
suggest things that would be worth adding to master (even if the original 
author didn't think of them as appropriate for master, or loses interest 
in them), for example.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Wed, 24 May 2023, Carlos O'Donell wrote:

> If we removed all the hooks, what is the consequence of a commit that is 
> wrong? Can we just fix the sources with a subsequent commit? Add a 
> light-weight linting check in 'make check' that additionally reports 
> this?

The consequence, at least for GCC, is that other scripting that depends on 
certain invariants of commits falls over.  Mapping from SVN revisions to 
git commits relies on only a single commit matching a given From-SVN 
commit message pattern rather than such a commit message getting quoted in 
another commit message in a way that confuses the mapping.  ChangeLog 
generation falls over if there are various problems with commit contents 
or messages.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Brian Behlendorf]

On 5/24/23 11:13, Siddhesh Poyarekar wrote:
> On 2023-05-24 12:29, Konstantin Ryabitsev wrote:
>> [...]
>> GCC and other projects are sufficiently high targets that we should 
>> not trust
>> the infrastructure to be secure or admins to be above being bribed or 
>> forced
>> under duress.
>
> It looks like the question of who to ultimately trust, either the 
> gatekeeper committer (or two) who is the only person to have write 
> access to the repository, or the admin who manages the box.  The 
> gatekeeper committer could also fabricate commits and push to the 
> central repository in the same way.  In fact, the gatekeeper committer 
> could choose to do worse, like delaying (or declining to merge) 
> someones patches.

It doesn't seem like an equal comparison - it seems like down one path 
there is an audit trail by which regular (machine+human) processes can 
detect malfeasance, while down another path there's a greater 
opportunity for compromise that can't otherwise be easily checked.

If GNU Toolchain devs haven't developed a full threat model (that would 
include things like "gatekeeper delaying someone's patches") that would 
be nice, so that you can weigh approaches that solve some threats but 
not others, or a combination that maximizes coverage. Social attacks 
matter - from a security POV it's less "gatekeeper being lazy/a jerk" 
and more "gatekeeper being compromised by an attacker".

> Signed commits would be nice, but we're not there yet as a community.

It's not quite as easy as "hey just add -s to your git commands" but 
it's not far from that. I'm not a part of the TAC so I won't go further 
than to suggest that a migration, where there will no doubt have to be 
some adjustments to process and flow, to introduce other non-zero-effort 
but non-blocking changes that enhance overall security/integrity.

Brian

-- 
Brian Behlendorf
CTO, Open Source Security Foundation
bbehlendorf@linuxfoundation.org
Twitter: @brianbehlendorf

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Wed, 24 May 2023, Konstantin Ryabitsev wrote:

> This workflow has one important problem -- you must ultimately trust the
> integrity of your infrastructure. It would be very easy for a malicious actor

Hence the importance of limiting access to infrastructure - and limiting 
what pieces of infrastructure have access to what others (rather than the 
present arrangements where many different services run on the single 
system with the master repository and with permissions allowing to write 
to it).

> 1. they can override any server-side validations
> 2. they can disable email notifications

A commit entering the repository without appearing on the mailing list is 
itself suspect.  (It can happen, but it's very rare; maybe about once a 
year for GCC.  The only cause I know of is a bug in the commit email 
generation that makes it fall over when the diffs are not valid UTF-8.)  
In the GCC case the commit emails include sequential numbers since the 
last release branched off; the motivation of those numbers isn't to make 
missing emails easy to spot (rather, they help with Jakub's system for 
speeding up bisection by having a large number of prebuilt compilers from 
different past revisions), but they do make missing emails easy to spot as 
a side effect.

> This situation could be partially mitigated if commits were cryptographically
> signed, but they don't appear to be.

It's quite possible commit signatures would be a useful thing to introduce 
in future.

> > Anyone with commit access can create their own branches in their 
> > refs/users/ namespace if they wish, and use them without needing review 
> > for commits going there.  Shared development branches under 
> > refs/heads/devel/ have whatever rules are established by the people 
> > setting up those branches.
> 
> From the git backend perspective, this is not optimal, because the resulting
> packs contain extra objects that aren't interesting to most cloners. For
> example, if I only leave refs/heads and refs/tags in the gcc repository, I get
> rid of over 1.1 million extra objects (from 3.9 million to 2.8 million) and
> the pack shrinks from 1.8GB to 1.1GB.

See what I said in the GCC service analysis about use of delta islands to 
make this efficient.  The configuration there is:

[pack]
        island = refs/heads/
        island = refs/tags/
        island = refs/(vendors/[^/]+)/
        island = refs/(users/[^/]+)/
        island = refs/(git-(svn-)?old)/
        island = refs/(dead)/
        island = refs/(deleted)/
        island = refs/(meta)/

(I don't think the other projects do anything this complicated with refs 
that aren't fetched by default, only GCC.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Wed, May 24, 2023 at 02:13:53PM -0400, Siddhesh Poyarekar wrote:
> It looks like the question of who to ultimately trust, either the gatekeeper
> committer (or two) who is the only person to have write access to the
> repository, or the admin who manages the box.

The maintainer's action are in the public, while the admin's actions are
hidden.

> The gatekeeper committer could also fabricate commits and push to the
> central repository in the same way.

1. Not with nearly as much impunity, especially if non-fast-forward pushes are
   disallowed. 
2. At best, a maintainer can modify someone's patch to introduce a deliberate
   bug, not freely modify a commit somewhere in recent history to add
   significant chunks of code like an admin can.
3. If it's someone else's code they are committing, the original developer
   will probably discover any malicious edits, because they will be watching
   their own commits show up in the repository.
4. It's much easier to audit and review the actions of 2-3 co-maintainers than
   when hundreds of people have write access to the repository.
5. When only a small subset of people can push to a repository, maintaining a
   valid keyring for verifying their cryptographic signatures is much easier
   than when 400+ people all have write access.

I could go on for quite a while still. My recent talk at the Linux Security
Summit was on the subject of how easy it would be to sneak a backdoor into the
Linux kernel where I go into many of these considerations (unfortunately, the
talk not yet posted on the LF channel).

> In fact, the gatekeeper committer could choose to do worse, like
> delaying (or declining to merge) someones patches.

The gatekeeper is accountable to the community and can be replaced. The
admin's actions may never be discovered or they can be blamed on someone else.

-K

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Wed, 24 May 2023, Konstantin Ryabitsev wrote:

> 2. At best, a maintainer can modify someone's patch to introduce a deliberate
>    bug, not freely modify a commit somewhere in recent history to add
>    significant chunks of code like an admin can.

Modifying an already-visible commit will be very obvious when anyone who 
has the original commit tries to pull, given that there should never be 
non-fast-forward pushes (to master, release branches or shared development 
branches) so people pulling (without local changes on the branch being 
pulled to) expect a fast-forward update.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 14:44, Joseph Myers wrote:
> On Wed, 24 May 2023, Siddhesh Poyarekar wrote:
> 
>> There's probably a viable middle ground here for glibc, where commits that
>> don't have an entry in patchwork get sent to the list.  Given the current
>> traffic for libc-alpha, it won't add much overhead to libc-alpha and also meet
>> auditing requirements.
> 
> I think this is overly complicating things.  (Though it would be good for 
> glibc to have the check done for GCC that once a commit is in the 
> repository, emails aren't sent out again for a merge of that commit into a 
> user / vendor / development branch.)
> 
>> For the release/* branches, it probably makes more sense to send the commits
>> to libc-stable, thus removing the need for developers to send [committed X.Y]
>> to the list.  This should then limit the utility of glibc-cvs to private
>> branch pushes.  I doubt if anybody cares about pushes to private branches, do
>> they?
> 
> I think commits to user / vendor branches can be of interest.  They may 
> suggest things that would be worth adding to master (even if the original 
> author didn't think of them as appropriate for master, or loses interest 
> in them), for example.
 
And in fact this is how I track Google's public work on the GRTE glibc branches.

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 14:46, Joseph Myers wrote:
> On Wed, 24 May 2023, Carlos O'Donell wrote:
> 
>> If we removed all the hooks, what is the consequence of a commit that is 
>> wrong? Can we just fix the sources with a subsequent commit? Add a 
>> light-weight linting check in 'make check' that additionally reports 
>> this?
> 
> The consequence, at least for GCC, is that other scripting that depends on 
> certain invariants of commits falls over.  Mapping from SVN revisions to 
> git commits relies on only a single commit matching a given From-SVN 
> commit message pattern rather than such a commit message getting quoted in 
> another commit message in a way that confuses the mapping.  ChangeLog 
> generation falls over if there are various problems with commit contents 
> or messages.
Thanks, both of those seem like they are surmountable challenges:

(a) The community may choose not to support SVN/CVS legacy repositories.

(b) You already need a ChangeLog post-process to fix issues, so the hook is
    a belt-and-suspenders way to reduce the work done by global maintainers to
    keep the ChangeLog clean.

Just a reminder that we are talking about the cost of the glibc migration, but have
also started discussing the gcc migration which is more complex.

(1) CVS/SVN read-only repositories.

If the GCC community decides not to keep the read-only CVS and SVN repositories in
favour of the git repository with all the currently converted history, then does
this remove some of the more complex git hooks?

(2) Changelog invariant checking.

ChangeLog generation as I understand it is a concatenation of the commit message
ChangeLog snippet (Nightly update_version_git), and already would need manual editing
if someone makes a mistake e.g. 32582c190b91287a0d1239162333f5ad0ac81e72 in gcc where
Jakub adjusts the entries? Does dropping this hook create more work for you and Jakub
and Richie to udpate malformed ChangeLog entries for the day? Who gets email if the
update fails?

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Joseph Myers]

On Wed, 24 May 2023, Carlos O'Donell wrote:

> (1) CVS/SVN read-only repositories.
> 
> If the GCC community decides not to keep the read-only CVS and SVN 
> repositories in favour of the git repository with all the currently 
> converted history, then does this remove some of the more complex git 
> hooks?

No.  The point of logic that looks for From-SVN: (and thus requires no new 
commits to have a line in the commit message starting From-SVN:) is 
precisely to convert from an SVN commit reference in e.g. an old email or 
bug report to a corresponding git commit, *without* referring to the 
read-only SVN repository.  I'm expecting the read-only repository to be 
made available as a downloadable tarball/dump rather than continuing to be 
served with the SVN protocol.

> (2) Changelog invariant checking.
> 
> ChangeLog generation as I understand it is a concatenation of the commit 
> message ChangeLog snippet (Nightly update_version_git), and already 
> would need manual editing if someone makes a mistake e.g. 
> 32582c190b91287a0d1239162333f5ad0ac81e72 in gcc where Jakub adjusts the 
> entries? Does dropping this hook create more work for you and Jakub and 
> Richie to udpate malformed ChangeLog entries for the day? Who gets email 
> if the update fails?

Manual work is rarely needed, generally only if there is a bug in the hook 
that means the hook fails to reject a commit but the nightly ChangeLog 
generation falls over (see the total of seven commits listed in 
git_update_version.py:IGNORED_COMMITS that resulted in such issues).  
Almost all the time when someone makes a mistake in the commit message 
formatting that would affect that nightly generation, it's caught by the 
hook and never gets into the repository.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-24 15:06, Konstantin Ryabitsev wrote:
> On Wed, May 24, 2023 at 02:13:53PM -0400, Siddhesh Poyarekar wrote:
>> It looks like the question of who to ultimately trust, either the gatekeeper
>> committer (or two) who is the only person to have write access to the
>> repository, or the admin who manages the box.
> 
> The maintainer's action are in the public, while the admin's actions are
> hidden.
> 
>> The gatekeeper committer could also fabricate commits and push to the
>> central repository in the same way.
> 
> 1. Not with nearly as much impunity, especially if non-fast-forward pushes are
>     disallowed.

If an admin makes a change that needs a fast-forward, anybody trying to 
sync their repos would see the forced update.  I'll admit though that 
it's arguable how many would actually notice the message.

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Siddhesh Poyarekar]

On 2023-05-24 14:55, Brian Behlendorf wrote:
>> Signed commits would be nice, but we're not there yet as a community.
> 
> It's not quite as easy as "hey just add -s to your git commands" but 
> it's not far from that. I'm not a part of the TAC so I won't go further 
> than to suggest that a migration, where there will no doubt have to be 
> some adjustments to process and flow, to introduce other non-zero-effort 
> but non-blocking changes that enhance overall security/integrity.

The challenge with cryptographic signing of commits is not a technical 
one, there are folks who oppose it on an ideological basis.  It's 
probably not that hard to drive consensus in the glibc community for 
this, but that's still going to be an additional hurdle to cross.  I'm 
not sure if the initial migration is the right time to tackle this.

Likewise for a process change that requires a single trusted gatekeeper. 
  Even if it has technical merit (I'm still not a 100% convinced), it's 
going to be harder to bring in a non-trivial process change like that 
during the initial migration.

Thanks,
Sid

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 17:27, Siddhesh Poyarekar wrote:
> On 2023-05-24 14:55, Brian Behlendorf wrote:
>>> Signed commits would be nice, but we're not there yet as a
>>> community.
>> 
>> It's not quite as easy as "hey just add -s to your git commands"
>> but it's not far from that. I'm not a part of the TAC so I won't go
>> further than to suggest that a migration, where there will no doubt
>> have to be some adjustments to process and flow, to introduce other
>> non-zero-effort but non-blocking changes that enhance overall
>> security/integrity.
> 
> The challenge with cryptographic signing of commits is not a
> technical one, there are folks who oppose it on an ideological basis.
> It's probably not that hard to drive consensus in the glibc community
> for this, but that's still going to be an additional hurdle to cross.
> I'm not sure if the initial migration is the right time to tackle
> this.
> 
> Likewise for a process change that requires a single trusted
> gatekeeper.  Even if it has technical merit (I'm still not a 100%
> convinced), it's going to be harder to bring in a non-trivial process
> change like that during the initial migration.

Agreed.

We should avoid contentious issues and cleanup only technical debt.

e.g.

* What are all the git hooks we use?
  * Which can be moved to webhooks?

* Stop supporting CVS/SVN and export tarball dumps.

* Migrate from moinmoin to readthedocs/sphinx.

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 17:12, Siddhesh Poyarekar wrote:
> On 2023-05-24 15:06, Konstantin Ryabitsev wrote:
>> On Wed, May 24, 2023 at 02:13:53PM -0400, Siddhesh Poyarekar wrote:
>>> It looks like the question of who to ultimately trust, either the gatekeeper
>>> committer (or two) who is the only person to have write access to the
>>> repository, or the admin who manages the box.
>>
>> The maintainer's action are in the public, while the admin's actions are
>> hidden.
>>
>>> The gatekeeper committer could also fabricate commits and push to the
>>> central repository in the same way.
>>
>> 1. Not with nearly as much impunity, especially if non-fast-forward pushes are
>>     disallowed.
> 
> If an admin makes a change that needs a fast-forward, anybody trying
> to sync their repos would see the forced update.  I'll admit though
> that it's arguable how many would actually notice the message.

Right. Some of us would, and review the situation.

Let me be clear though, Google *accidentally* force pushed on glibc master branch
on 2023-01-10 and the hooks didn't stop them, and it is a known bug in our hooks.

Joseph noticed and called it out... I didn't notice. Nobody else noticed because
it was the most recent thing in HEAD and nobody had pulled yet.

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/24/23 17:53, Ian Kelling wrote:
> 
> Carlos O'Donell <carlos@redhat.com> writes:
> 
>> On 5/22/23 10:37, Ian Kelling wrote:
>>>
>>> GNU already has a system for release uploads to https://ftp.gnu.org that
>>> works very well and I help maintain it.
>>
>> Agreed, and I use it for glibc to upload release tarballs.
>>
>> However, the tarball release process is quite heavy-weight for release branch maintenance.
>>
>> In addition the downstreams sync from git release branch -> distro git to get the 
>> rolling release updates.
>>
>> So all the other discussions about trusting git become relevant here.
> 
> Yes, that all makes sense to me.

Just for clarity, and quoting again from my service evaluation for glibc:
~~~
* Release tarballs (ftp upload of gpg-signed release tarballs)
  https://ftp.gnu.org/gnu/libc/
  * Use gnupload script to gpg sign uploaded tarballs.
   * Uses ncftpput to place files into /incoming directories.
   * Network ftp access required.
  * Managed by the GNU Project/FSF.
  * Migration notes:
   * No migration required, would continue to upload to FSF.
   * Longer term discussion to use something more advanced.
   * Though good to have a backup following kernel best practice.
~~~

I noted that no migration is required here, but a backup has value.
If the backup, kput, used my ssh key via gitolite then it's already
using something I have setup as a maintainer to access git.

Several of the GNU Toolchain projects today upload to sourceware.org
*and* gnu.org. After the upload is complete we have mirroring from
other sources. So only the service of "upload" during a release window
could be improved by having an alternative.

The longer term discussion point there is specifically about what I
raised there, that release branch maintenance via tarballs is much
heavier than needed. Upstream uses git. Downstream uses git. And we
want a git to git mirror-style setup. In Fedora we would call this
"source-git" (as opposted to "dist-git" which contains rpm spec
files and configuration information). Today we sync Fedora Rawhide
dist-git weekly from upstream glibc git in order to have weekly
CI/CD. This conversation is something I can have with the GNU Project
and attempt to document best practice.

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Ian Kelling]

Carlos O'Donell <carlos@redhat.com> writes:

> On 5/22/23 10:37, Ian Kelling wrote:
>> 
>> GNU already has a system for release uploads to https://ftp.gnu.org that
>> works very well and I help maintain it.
>
> Agreed, and I use it for glibc to upload release tarballs.
>
> However, the tarball release process is quite heavy-weight for release branch maintenance.
>
> In addition the downstreams sync from git release branch -> distro git to get the 
> rolling release updates.
>
> So all the other discussions about trusting git become relevant here.

Yes, that all makes sense to me.

- Ian

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

On 5/22/23 10:37, Ian Kelling wrote:
> 
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
> 
>> In the case when we can't use the git tag note process, we can also do tarball
>> uploads using our own "kup" tool that is done over ssh and uses the gitolite
>> permission framework. I'm quite shocked that someone in the world still uses
>> FTP for any uploads. :)
> 
> GNU already has a system for release uploads to https://ftp.gnu.org that
> works very well and I help maintain it.

Agreed, and I use it for glibc to upload release tarballs.

However, the tarball release process is quite heavy-weight for release branch maintenance.

In addition the downstreams sync from git release branch -> distro git to get the 
rolling release updates.

So all the other discussions about trusting git become relevant here.

-- 
Cheers,
Carlos.

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Ian Kelling]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> In the case when we can't use the git tag note process, we can also do tarball
> uploads using our own "kup" tool that is done over ssh and uses the gitolite
> permission framework. I'm quite shocked that someone in the world still uses
> FTP for any uploads. :)

GNU already has a system for release uploads to https://ftp.gnu.org that
works very well and I help maintain it.

Debian uses ftp for package uploads according to
https://www.debian.org/security/faq and of course they support package
downloads over ftp too. There is nothing wrong with it for our use
case. Ssh would have the disadvantage of managing extra keys, but I'm
not opposed to adding it if it would make sense. If there are any
features lacking for the GNU ftp, we would be happy to add them. For GNU
maintainers to discuss potential improvements to the ftp the best place
is https://lists.gnu.org/mailman/listinfo/gnu-prog-discuss. There are
also multiple inherent advantages of GNU packages to use the GNU release
site. For example, for any changes to the set of keys gpg authorized to
upload, there is a manual review by FSF tech team and GNU volunteers.

-- 
Ian Kelling | Senior Systems Administrator, Free Software Foundation
GPG Key: B125 F60B 7B28 7FF6 A2B7  DF8F 170A F0E2 9542 95DF
https://fsf.org | https://gnu.org

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Mon, Apr 03, 2023 at 01:24:21PM -0400, Carlos O'Donell wrote:
> That means the the next steps likely look like this (needs further discussion):
> * Iterate over the services below and discuss how and what to migrate them to.

Hello, everyone:

Sorry for continued delays -- I had a few false-starts and plenty of
usual distractions, but I am finally able to move on this task with a
clearer picture of what I am doing. I know you asked for costs, but they
are going to depend on the finalized set of services, so I am going to
set this aside just for the moment and focus on the technical aspects of
the migration. It's not really that I am keeping anything close to heart, it's
just that a lot of this infrastructure will be using shared deployments, and
it's as not straightforward to easily assign dollar figures to cycles as cloud
hosting setups would like us all to believe. :)

# Service enumeration

## Mailing lists

The number of mailing lists doesn't really matter -- they are all managed via
mlmmj and public-inbox and we're only really looking at the general volume of
messages and size of archives.

Looking at your list, the only thing we don't do is clamav, because it's
mostly security window-dressing. Your client-side antivirus and end hosting
provider is much more likely to do a good job catching any malicious content
than clamav.

The mailing lists will be using shared infrastructure (subspace.kernel.org)
for the actual mailing lists, and for the public-inbox archives we can either
host them as part of lore.kernel.org, or deploy a separate service just for
your own list -- your preference. There are upsides and downsides to either
choice here.

Our mailing list infrastructure is uniquely tailored to public-inbox users,
with messages being written to the archives *before* they are even sent out to
subscribers, which helps speed up access for users who aren't using the
traditional "push" subscription module, but are pulling search-based results
instead (using lei). This proves to be a much more reliable mechanism for
patch-based workflows than traditional SMTP push subscriptions.

## Bugzilla

Everything on your list looks fine, and we can certainly replicate these
features. We have also recently released our own experimental bugbot that
allows us to much better integrate public-inbox managed mailing lists with
bugzilla and git:

https://lore.kernel.org/workflows/gmnswy45efztvcyc3xqymm52h7cdfwjz64go2wy3zyeqcqwb76@36fva4hyip4q/

The major improvement of this vs. bugzilla's native email interface is that
bugs can be created from mailing list discussions, not just the other way
around. The bugbot will need to be slightly adjusted for CTI needs, but
otherwise I think it would be a better solution than Bugzilla's native email
integration. For example, when bugmail is sent to the mailing list and someone
responds who does not have a Bugzilla account, their account would be
auto-created. Our approach does not require this and will happily allow fully
two-way bridging between Bugzilla and mailing list discussions.

## Git

We are using Gitolite for all our projects, and we strongly suggest doing the
same here. It would be straightforward to migrate from Unix user/group
permissions to Gitolite authN, so I do not expect this to be complicated or a
showstopper.

The rest of it will need some work to figure out the best way forward:

* Sending per-commit email to mailing lists -- this seems like a vestigial
  feature from the pre-git past. Does it really serve any purpose? I am
  fighting to kill a similar feature used on the kernel.org side, because I do
  not see any remaining legitimate use for it -- it just creates a lot of mail
  traffic that nobody really reviews.

* We do not currently allow direct hooks on the server side, because running
  arbitrary scripts with the permissions of the git server is a very bad
  anti-pattern when it comes to repo security. It also tends to make pushes
  super slow and frustrating to developers, especially those working on slow
  or unstable connections that can go rapidly stale, resulting in failed
  pushes.

  So, we will need to wrap our collective heads around what your hooks are
  doing now and implement them via alternative means. For example:

  * denying force-pushes to specific branches (rebase/non-fast-forward
    updates) is a native feature of Gitolite
  * our bugbot is able to integrate git commits with bugzilla better than
    sending out mail to an bugzilla ingestion point, and this doesn't need to
    happen on commit (we use git log queries)
  * the same can be done for IRC/Matrix notifications, if that is still
    desired

  The only thing that would require more work is style checkers and other CI
  functionality that really should not be happening on-push. I suggest that
  this moves entirely to patchwork CI, discussed below.

## Patchwork

Everything described under "Patch management" we can do, with some tweaks. For
example, we feed patchwork from public-inbox archives instead of directly
piping incoming email. We also have our own git-patchwork-bot that is able to
auto-close patches and series when it seems them show up in git repositories,
with follow-up messages to authors and to the mailing list.

We also have multiple projects using patchwork CI integration (e.g. see
https://patchwork.kernel.org/project/netdevbpf/list/), which is our
recommended way of doing pre-commit checks on patch series.

## Wiki

I strongly believe that wikis are quickly fossilizing creatures of a bygone
era. If at all possible, we will not migrate any wikis, instead migrating them
to readthedocs-style documentation (e.g. using pandoc for migrations). We have
done it with a number of projects now and will be happy to assist in this case
as well.

There are multiple obvious upsides:

* Fast pre-rendered documentation
* Sane site organization
* No more dealing with wiki spam
* CI integration for actual style-enforcement

I do not believe there is any reason to continue to run wikis in 2023.

## Project CI

I am side-stepping this for now, because CI was outside the scope of our
discussion. If that has changed and we're now making CI a core part of the
migration, then I will need to loop in our Release Engineer team for their
analysis and suggestions.

## Websites

Any static-site generator solution will do, including readthedocs as with the
suggested wiki migrations. We have previously used Jekyll and Pelican.

## Release tarballs

There are multiple ways of getting this done, including completely automated.
For example, stable kernel releases are generated server-side by using and
verifying the PGP signature found in a git note attached to the release tag.
This doesn't work if the tarball is not directly generated from a git
repository (for example, if config scripts must run first).

In the case when we can't use the git tag note process, we can also do tarball
uploads using our own "kup" tool that is done over ssh and uses the gitolite
permission framework. I'm quite shocked that someone in the world still uses
FTP for any uploads. :)

## IRC

We don't run our own IRC servers, but LF may potentially offer Matrix, which
we may bridge to an IRC server of your choice (and if needed).

# Bird's eye infrastructure overview

To implement everything as discussed above, we would deploy a small subset of
VMs:

- gitolite VM, which is the primary of all your git repositories and tarball
  releases
- bugzilla VM, which hosts the bugzilla itself
- bot VM, which runs all the integration bots, such as bugbot,
  git-patchwork-bot, patchwork mail ingestion bot, site/documentation build
  bots, etc -- basically all infra "CICD"
- SQL VM for bugzilla, patchwork, and bot databases
- patchwork container (since it's a Django app, we don't need a VM here)
- public-inbox archive frontend VM (if not using lore.kernel.org)

The price footprint on these VMs and containers is fairly small, especially
because we will try to shift most high-traffic services to shared
infrastructure:

- public git repositories will be replicated to multiple worldwide frontends,
  same as git.kernel.org
- all static sites will be served from shared infrastructure
- tarball releases will be replicated and served via mirrors.kernel.org

Furthermore, once other toolchain projects start migrating, we can reuse all
of this infrastructure for that purpose.

# Next steps

Before we go any further and start talking concrete VMs and costs, we need to
figure out the CI part that I side-stepped, and the complete rework of git
hooks that I suggest needs to happen before we can consider switching over. I
do feel very strongly that none of it should be running on git push -- but
without a deeper understanding of your needs, I cannot rule anything
completely out. Should we start a separate thread to make sure that I'm not
suggesting anything that isn't workable because I'm not fully grasping your
needs?

# OpenSSF

Lastly, I do believe that CTI migration is already partly or fully covered by
OpenSSF grants. I am in a separate conversation with folks from the Alpha
Omega group at OpenSSF to better understand their goals and the longevity of
their grants -- but this will probably also impact the financial side of the
equation (I just don't know to what degree, yet).

Again, apologies for the continued delays. Please know that CTI is currently
front and center in my focus, so I hope to catch up on a lot of tasks that
have been on the backburner.

Best regards,
Konstantin

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Brian Behlendorf]

On 4/12/23 13:46, Konstantin Ryabitsev wrote:
> On Mon, Apr 03, 2023 at 01:24:21PM -0400, Carlos O'Donell wrote:
>> Konstantin,
>>
>> The GTI TAC met on 2023-03-08 (please ignore the date typo 06-08):
>> https://lore.kernel.org/gti-tac/b72a0c4f-fed3-7418-e9d6-7a277bf37b3e@redhat.com/
>>
>> There was consensus that we should start working through the process steps one
>> project at a time and pipe-clean the process and learn as we go.
>>
>> I volunteered glibc as the first project because I know the most about the project
>> and the requirements.
>>
>> I am providing the full list of glibc services for review and pricing by LF IT.
> Carlos:
>
> I'm not ignoring you, it just so happened that the ice storm (and the
> subsequent loss of power) + Easter holidays basically ate away a whole week of
> my productive time. I'll be responding to your email shortly, apologies for
> the delay.
>
> -K
>
OMG! Wow, no problem and completely understood. Thanks and good look 
with the inbox wrangling.

Brian




-- 
Brian Behlendorf
General Manager, Open Source Security Foundation
bbehlendorf@linuxfoundation.org
Twitter: @brianbehlendorf

Re: Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Konstantin Ryabitsev]

On Mon, Apr 03, 2023 at 01:24:21PM -0400, Carlos O'Donell wrote:
> Konstantin,
> 
> The GTI TAC met on 2023-03-08 (please ignore the date typo 06-08):
> https://lore.kernel.org/gti-tac/b72a0c4f-fed3-7418-e9d6-7a277bf37b3e@redhat.com/
> 
> There was consensus that we should start working through the process steps one
> project at a time and pipe-clean the process and learn as we go.
> 
> I volunteered glibc as the first project because I know the most about the project
> and the requirements.
> 
> I am providing the full list of glibc services for review and pricing by LF IT.

Carlos:

I'm not ignoring you, it just so happened that the ice storm (and the
subsequent loss of power) + Easter holidays basically ate away a whole week of
my productive time. I'll be responding to your email shortly, apologies for
the delay.

-K

Next steps from GTI TAC meeting on 2023-03-08 - Evaluate cost of glibc migration.

[Carlos O'Donell]

Konstantin,

The GTI TAC met on 2023-03-08 (please ignore the date typo 06-08):
https://lore.kernel.org/gti-tac/b72a0c4f-fed3-7418-e9d6-7a277bf37b3e@redhat.com/

There was consensus that we should start working through the process steps one
project at a time and pipe-clean the process and learn as we go.

I volunteered glibc as the first project because I know the most about the project
and the requirements.

I am providing the full list of glibc services for review and pricing by LF IT.

When I say "review" I am looking for a critical eye from LF IT where you might
recommend a better solution that you can support in the long term e.g. wiki,
bugzilla, mailing lists, mailing list archives, etc. Likewise for things we should
drop as legacy and not convert them over.

I expect that pricing depends on the final solution and the exact work to be done
so this will take some iteration. Likewise I expect pricing is split between NRE
for the shift vs. ongoing.

That means the the next steps likely look like this (needs further discussion):
* Iterate over the services below and discuss how and what to migrate them to.
* Start socializing with the glibc project community.
* Finalize on the approximate details of the migration and ongoing services.
* Finalize on the cost and timeline.
* Get the GTI TAC to review and approve.
* Get the GTI Board to review and approve.
* Take it to the glibc project for approval.
* Execute on timeline.

Complete set of known glibc services:

* mailing lists
  * Mailman 2 mailing lists: https://sourceware.org/mailman/listinfo/*
    * libc-announce
    * libc-alpha
    * libc-stable
    * libc-help
    * libc-locales
    * libc-testresults
    * glibc-cvs
    * glibc-bugs
    * glibc-bugs-regex (limited bugs just for regex).
    * Closed legacy mailing lists:
      * libc-ports: https://sourceware.org/mailman/listinfo/libc-ports
      * libc-hacker: https://sourceware.org/mailman/listinfo/libc-hacker
      * Also the older MHonArc archives (/legacy-ml/) - no longer updated, but the 
        /legacy-ml/ URLs and /ml/ redirects to them need to keep working.
        E.g. https://sourceware.org/legacy-ml/libc-alpha/2020-01/
        E.g. https://sourceware.org/legacy-ml/glibc-bugs-regex/2020-03/
    * Mailing lists accept non-html email only.
    * Run through spamassasin
    * Run through clamav
  * Pipermail archives:
    * https://sourceware.org/pipermail/*
    * e.g. https://sourceware.org/pipermail/libc-alpha/
  * public-inbox archives:
    * https://inbox.sourceware.org/*
    * e.g. https://inbox.sourceware.org/libc-alpha/
    * Not all inboxes work correctly e.g. glibc-bugs-regex doesn't work.
  * Migration notes:
    * Only supporting public-inbox for ml archives.

* bugzilla 5.0.4+
  * Uses backend SQL database of MariaDB 10.3
  * Must be able to send email to glibc-bugs mailing list.
    * Don't know how email is routed to this list.
  * Must also send email glibc-bugs-regex mailing list.
    * Don't know how email is routed to this list.
  * Must be able to send email to all users on the bug.
  * Must be able to receive email when someone responds to a glibc-bugs
    email e.g. sourceware-bugzilla@sourceware.org.
  * Custom Administration->Groups settings for User RegExp.
    * canconfirm: Allow certain domains to always be able to confirm bugs.
    * editbugs: Likewise but for editbugs.
  * Must have REST API enabled to allow RM to generate release list
    of fixed bugs using the glibc/scripts/list-fixed-bugs.py script
    e.g. https://sourceware.org/bugzilla/rest.cgi/
    * Implies that non-logged-in users can list and view all bugs
      that were fixed for the release.
  * Must have account creation disabled due to spamming.
  * Must have someone with Bugzilla admin access to:
    * Add new users to bugzilla.
    * Add new Product components, versions, and milestones.
    * Add new Key Words
    * Remove users.
  * Migration notes:
    * https://sourceware.org/bugzilla/
    * Consider starting fresh in new BZ instance and freeze old product.
    * glibc in old instance marked "Not open for new bugs."

* git 2.31
  * Allows per-user access to commit to the glibc repo.
  * Allows per-user access to commit to the legacy glibc-ports repo.
  * Uses group access to control repository access.
  * Must be able to send email to glibc-cvs mailing list with one
    email for each commit made by a developer to any branch of the repository.
  * AdaCore hooks need more thorough audit for required services.
    * Must be able to send email to bugzilla to update bugs.
      * Done by AdaCore hook 'file-commit-cmd'
      * Configured to use email-to-bugzilla-filtered command.
        * Uses connection to SQL database to determine if bug exists.
  * Currently uses shared AdaCore hooks configured via origin/meta/config 
    * Active hooks:
      * post-receive
        * AdaCore post_receive
        * /git/glibc.git/hooks-bin/post-receive
          * Triggers irkerhook.py (see notes below).
          * Does not work today, likely due to requirement to register OFTC user.
      * post-update
	* Standard git-update-server-info.
      * pre-receive
	* AdaCore pre_receive
    * AdaCore config:
      * No max line lengths.
      * Allow UTF-8 in commit messages.
      * 5MiB max email size.
      * Max 500 commit messages for larger commit series sent to glibc-cvs.
      * Reject merge commits to master and release branches.
      * Allow rebasing only private branches (non master and non release).
      * Run minimal style checker, nominally for whitespace issue rejection.
        * Run extra commit checking to avoid source address for author being wrong.
          * /git/glibc.git/hooks-bin/commit_checker
            * From email format checker. No special requirements.
        * /git/glibc.git/hooks-bin/style_checker
          * Style chcker. No special requirements.
      * Send email to bugzilla if a commit mentions a bug.
        * /git/glibc.git/hooks-bin/email-to-bugzilla-filtered
          * Uses /sourceware/infra/bin/email-to-bugzilla
          * Must be able to connect to bugzilla SQL database.
          * Does not appear to work today. We don't get emails for commits with bugs.
      * Send IRC message to per-project configured IRC channel.
        * Involves irkerhook.py and git config information for project.
        * Hook must be able to connect to external IRC networks to post IRC notices.
  * Migration notes:
    * Allow community to manage access?

* wiki
  * Uses MoinMoin 1.9.10
  * Must have account creation disabled due to spamming.
    * Uses EditorGroup permissions to allow any community member to add a new
      community member to the wiki e.g. human vetting another human.
  * Must be able to send notification emails.
  * Cron run to purge users not in EditorGroup to prevent wiki slowdown.
  * Migration notes:
    * Preference for something git based.

* patch management.
  * Uses patchwork v3.1.1.post18-g11cf1f3
  * Must be able to receive email (as part of collecting patch data)
  * Must be able to send emails as part of account verification.
  * Uses django for administration
  * Must allow authenticated REST API access for patchwork.
    * Currently rate limited.
    * Used by SLI tools (Carlos O'Donell)
      * Run manually on developer systems.
    * Auto-close on commit patchwork bot (Siddhesh Poyarekar)
      * Run on sourceware.org via cron.
  * Used for weekly patch management meetings.
  * git-pw integration used to access patchwork directly using REST API and API token.
  * Migration notes:
    * Patchwork a strong requirement for upstream CI/CD.

* Red Hat Bluejeans remote meeting system.
  * Must allow remote video and audio for participants around the world.
  * Allows weekly glibc patch review meetings for patch review collaboration.
  * Meetings must operate without host needing to be present so community can host.
    * Delegating host is difficult in bluejeans.
  * Managed by Bluejeans/Verizon.
  * The glibc community has switched to LF BBB instance for the last 24 meetings.
    * Since 2022-12-12 we have been using LF BBB instance successfully to host weekly meetings.
  * Migration notes:
    * Already migrated to LF BBB. NOP.

* pre-commit CI system.
  https://gitlab.com/djdelorie/glibc-cicd
  * Run inside a VM.
  * Uses networkless containers for further build isolation.
  * Highest risk system because it runs mailing list posted patches.
  * Event curation system (curator):
    * Must have network access to patchwork REST API.
    * Must have access to SQL database for storing state.
      * Currently using MariaDB.
    * Must allow runners to access curatore REST API URL.
    * One curator currently hosted by DJ Delorie.
  * Event running system (runner + trybots):
    * Must have network access to curator REST API.
    * Must have local network access to rabbitmq queue (job delegation)
    * trybots must have local network access to rabbitmq.
      * Must have network access to patchwork REST API to post results.
      * Must have network access to container registries to pull modern containers.
      * Must have network git access to pull updated glibc git repo.
    * Generally the runner and trybots are on one site together.
      * Avoid passing rabbitmq traffic beyond the local network.
      * Eventual emailing of results to the mailing list will happen via another bot
        that is distinct from this system to avoid the runners needing anything but
        restricted network access.
    * One runner hosted by DJ Delorie	
    * One i686 trybot hosted by DJ Delorie
    * One "patch applies" trybot hosted by DJ Delorie
  * Migration notes:
    * Could argue no migration required. LF hosts patcwork. Community hosts try bots.

* Website (sourceware.org)
  * CVS hosted website.
  * Static redirect to gnu.org website.
  * Migration notes:
    * Need something for a static site.

* Website (gnu.org)
  https://www.gnu.org/software/libc/
  * CVS hosted website uploads along with manual.
    * Manuals are generated with scripts in the CVS repo and generated files committed.
  * All static content.
  * Website automatically updated after CVS commits.
  * Manged by the GNU Project/FSF.
  * Migration notes:
    * No migration required see previous "Website (Sourceware.org)" entry.

* Release tarballs (ftp upload of gpg-signed release tarballs)
  https://ftp.gnu.org/gnu/libc/
  * Use gnupload script to gpg sign uploaded tarballs.
   * Uses ncftpput to place files into /incoming directories.
   * Network ftp access required.
  * Managed by the GNU Project/FSF.
  * Migration notes:
   * No migration required, would continue to upload to FSF.
   * Longer term discussion to use something more advanced.
   * Though good to have a backup following kernel best practice.

* Translation project services
  https://translationproject.org/html/welcome.html
  * https network access to TP servers to fetch uploaded translation files.
  * Managed by the Translation Project.
  * Migration notes:
    * Don't expect to have a replacement.

* IRC services on OFTC and Libera.Chat
  * Using #glibc on both networks for community interaction.
  * Migration notes:
   * Don't expect to have a replacement.

-- 
Cheers,
Carlos.